Tuesday, December 23, 2008

How to destroy your business 101 - network security


Throughout the course of my career, I've had opportunities to work with a range of companies both large and small. Along the way, I've seen some amazingly bad practices. The root cause of each of these scenarios is usually a mixture of inexperience and/or laziness. I'm not a network expert, but I know a bad idea when I see one. Here are some example of grievous holes in security that I've run into first hand.

Shared Accounts
Don't do it. I've seen this manifest itself in few different ways. At one shop, every member of the IT staff knew the password to the domain admin account. This is bad. The IT manager didn't realize that you could add accounts to the Domain Administrators Group. *sigh* The thing is, if a bunch of people have access to the same account, it's almost impossible to tell who screwed something up. Also, if it's hard to tell who dunnit, it's more likely that somebody is going to do it. In other words, if you have a disgruntled employee in house that has domain admin access and you have no way to audit access, he or she probably knows it and is free to wreak whatever havoc he or she wants to without much fear of getting caught.

More recently, I saw the same problem from a different angle. Instead of a bunch of people having the password to the domain admin account, everybody had the username / password to a test account that is part of the domain admin group. Oh snap. With these credentials, it would be possible to wreak devastation on the network with no trail. This isn't a good. The first thing I did was report this huge hole in security, so I'm hoping that somebody does something. It's not my responsiblity, but I'd rather not have to twiddle my thumbs as the IT team rebuilds the servers.

Easy passwords for sensitive accounts
This was a legendary scenario. Most companies use VPN to provide remote access to the corporate network. The passwords for these accounts are usually required to have strong passwords at the the very least. The robust solution should require certificates, SecureID fobs, etc. I stumbled onto this one night when I was bored at home and was curious if I could guess the password to one of the salepersons accounts. I fell out of my chair when I got it on my second guess. Oh yeah, it was "password".

Note that this was a large company, with over 300 employees that did many millions of dollars worth of business. Oh, and it was a medical supply company. Holy crap. That was a huge violation of HIPAA rules. I called the IT manager to fix the problem immediately. He was worried that he would get support calls from the sales staff in the middle of the night so he didn't want to deal with it until the morning. I knew that I would be the one fixing the problem if the network got comprimised, so I broke protocol and disabled the VPN server for the entire company. I don't like sidestepping procedure but in this case I felt I had no choice. The first thing in the morning I let the executive meeting what was happening. Fortunately, the exec team understood the gravity of the matter and we resolved the issue pretty quickly.

Wireless Access Points
In another case, I was walking around the building I worked in. While walking by the office of the IT manager, I noticed that there was a wireless router plugged into the network. Snap. I asked him what it was for and he stated that we was playing with it, but it was no big deal because it didn't have a big range.

This was at the same company (and yes, the same IT manager). Did I mention the gobs of patient data that we were responsible for? I advised him that he probably should not be plugging that into the network. He eventually saw my point, but I was amazed at the complete lack of forethought.

There are ways to safely incorporate wireless networking into an enterprise architecture, but the right solution usually involves several layers of security including encryption, access control and firewalls. Just plugging a cheap consumer wireless router into your office wall with default settings (wide open access) is a recipe for disaster.

Mystery Machines
At one post, I noticed there were several servers that were hanging out in the corner of the server room. They weren't labeled and nobody had any idea what they did. Better yet, nobody had the root password to the machines. Holy crap!

Basically, an intern had set up some servers for the company a while back, but never shared the passwords. Out of fear, the IT manager refused to turn off the machine. Later, when I became responsible for the network, the first thing I did was unplug those machines. It turned out they were service some utility, but weren't critical. I instructed the IT staff to recycle the machines with a fresh OS install so that we knew exactly what was installed.

Finally
There are many subtle ways in which network security can be comprimsed. These were some gaping holes that were just asking for trouble. Take an inventory, if you're doing any of the above - stop!

Friday, December 19, 2008

Automation

I get bored easily. I wouldn't make a good factory line worker because I'm horrible at repetition. Once I've figured something out or gotten proficient at something, I lose interest for the most part (this excludes multiplayer first person shooter games - I can play those forever). If I had to choose between working on an assembly line or wrestling rabid alligators, I would probably say something like "How big are the gators?"

This isn't necessarily a bad thing (or at least that's what I tell myself). When it comes to building software, I like to design and code, improve process - that sort of thing. Doing repetitive things like running database scripts or running reports on project status are Kryptonite to me. I would rather spend a few days building a tool to do the work for me than do it by hand every day / week, etc.

What are good candidates for automation? Pretty much anything that is repetitive.

Daily, Weekly, Monthly, Reports.
Do whatever it takes to make these pushbutton. This doesn't apply to one off reports. However, if you're spending a bunch of time copying and pasting between Excel spreadsheets, you're doing something wrong. If you automate the reporting process, any time spent daily is tweaking. Tweaking is important for reporting. If you're expending all of your energy just perfoming the mechanics of assembling the report, you're not as likley to make improvements to the report itself.

Builds.
Find a way to automate the build. This is a continuous integration sort of thing, but it's a productivity enhancer. Broken builds make broken teams. Can you live without it? Sure, on a small team it isn't too bad and I've worked that way plenty. However, on a larger team, I can't imagine living without it.

Installers

If you're using the Setup project type in Visual Studio, it's pretty easy, just build the setup project from the IDE. If you're using install shield, find a way to script it / run it with nmake, etc so that you don't need the "install guy" (who is probably a coder responsible for a bunch of other things) to take time to "turn the build" and make an installer.

I ran into an extreme version of this at one shop where the build took something like 14 man hours to turn. This one guy would go back to the build machine (it was the only environment where the build would work) in the closet and do his thing. He eventually figured out how to script the whole thing with make files, but the manual process was ridiculous.

Odd bit
There are plenty of opportunities. While working at one place, the shiny new web server got set up just before the IT guy left for vacation. Genius. Of course, the site developed all sorts of problems and was going down every few minutes.

We fixed the (numerous) issues and put it back into production. This one guy was sitting there hitting the web site every few seconds to make sure that it was staying up. I threw together a quick app that monitored the site and would make noise if the site went down. He appreciated the help but was like "You must have too much time on your hands." My response was "that's how I make time." I try to invest a little bit of time up front so I don't waste time later. It's a net gain.

Thursday, December 11, 2008

Starting a new Dev project


It's a glorious day. You have the green light to start a new development project. You can almost feel your hands on the keyboard typing out glorious code and coming up with fresh frameworks that are devoid of all that legacy that has been keeping you down. Not so fast! There are a lot of critical things that have to be in place for a project to be successful. Here are some lessons I've learned from participating in the development of a number of projects.

Process
Pick a process. It's 2008 for crying out loud. Developing software without a process is just plain silly. I've been a big fan of the iterative approach including Scrum. Can you succeed with waterfall? Sure. Even a blind squirrel finds a nut every now and then. There's a reason why most people cringe when they hear the word "waterfall." Even the creator of the process sites it as a generally bad idea.

There are some different approaches, including Spiral Development and Evolutionary Prototyping. You can also go crazy and implement CMMI. I haven't used that particular model, but from what I hear it's pretty involved. If you haven't developed using a software process before, I probably wouldn't start there.

Requirements
Yup, you need them. There's no free lunch here. What is there to justify? This lesson has been learned over and over and over again. Without a clear list of items to complete, nothing good will come. Go down to a pretty low level of detail. Items such as "Options Dialog" are very high level. You should go down to the level of "The user shall be able to select a default font for the document. " The more work and detail you do at this point in the game makes the rest of the project go much, much faster.

Prototypes
Prototypes are critical in proving technologies and approaches. Within the source of a project I end up creating a number of prototypes. Some of these are little one off sandboxes that I toss. However, with most of these prototypes, there is some value in saving these as examples for other developers on the project, or as an easy way to capture the knowledge gained while sandboxing for later. I usually create a folder in source control just for prototypes.

Of course, if you're using the Evolutionary Prototype process, prototypes are bakes in. If not, prototype each new piece of technology before adding it to the main source tree. It's worth the time and will make sure that what you add to the main source tree is of decent quality.

Also, if you're team is learning a new language for this new project, plan on throwing away the first attempt at building the product. The only way to learn how to code is by doing it, but assume that the first iteration of building whatever widget you're working on will be crap. You'll learn lots, but you'll need to rebuild it. The good news is that when you do rebuild it, it will take a fraction of the time and the quality will be several orders of magnitudes better.

Source Control
There are many options available for Source Control. Just don't use Source Safe. If you're thinking about using Source Safe, just shoot yourself in the foot. Literally. It will be as painful as trying to use Source Safe and will save time (including the hospital visit). It's slow, unreliable, and horrible. The lack of critical features such as atomic commits, not losing data, etc is unacceptable. I've only used Source Safe and Team Foundation Server (TFS), so I don't have any advice to share for specific systems outside of those. As far as TFS goes, there are pros and cons which I go into here.

Continuous Integration
Do this from the beginning. Make it easy for everyone to build the software. This is a great opportunity for automation. Different platforms have their own tools, but I'm a big fan of Team Build for TFS 2008. It's pretty easy to set up and does most of what I want (although gating isn't going to come until TFS 2010 - aka Rosario)

Installer
If you application requires an installer, build one from the beginning. As soon as you're past the prototype stage, build the installer and maintain it throughout the dev process. Doing months of work then throwing it over the fence to the installer guy is a recipe for disaster. The amazing thing is that I've seen this done more often than not.

If you build this upfront, you'll be able to get feedback from the stakeholder much sooner in the process. The shorter this feedback loop is, the less likely you'll be to stray too far away from what he / she / they expect.

Short Feedback Cycle
Well, we just covered this in the installer bit, but it's worth mentioning on its own. Schedule work in short enough increments (2 to 4/5 weeks seems to be the sweet spot range) that you can get feedback. That breaks your work down into manageable bits and provides specific opportunities for the stakeholders to view the product (whether it's internal or external).

Internationalization
Build it in from the ground up. Yes. Do it now. I go into it in more detail here.

Draw Pictures.
Draw lots and lots of pictures. Keep them around. Whiteboards are the best for brainstorming (although it doesn't work well remotely). Use Visio - it's great. I've used the flowchart diagramming to do all sorts of things from program flow to installation scenarios.

Have a big overview picture. It's good stuff. Hang it on the wall. Keep it up to date. It will help a lot of conversations to have the big picture to point at to talk about where problems are.

Testing
Do it. Do it from the beginning. Have a tester develop a test plan from the requirements. Testing in flight will save sooooo much time. Throwing a product over the wall to testing after months of coding is such a horrible idea it's staggering to know that there are still teams doing this. If you want to get your product to market as fast as possible, this facet of the development process has the greatest impact on delivery time (not to mention product quality which ultimately effects support costs).

Unit Tests
Yes do them. Write your unit tests before you implement your code. Having repeatable tests that you can run at the push of a button is magical. If you can hook up those unit tests to an automatic build process, that's even better.

Code Reviews
Code reviews are a stellar bang-for-the-buck proposition. It is by far the cheapest way to reduce errors and ensure that the teams coding standards are followed. This is also a good way to make sure that individual members of the team aren't reinventing the wheel. I don't know how many times I've looked at someone's code on the team and said something like "oh hey, we already have something that does that. Use it instead."

Code Reviews should start from the very beginning of the project. Most likely, the code reviews will be the longest in the beginning of the project while the team is deciding on best practices. As the project continues, the reviews should be done just as often, but they probably won't take nearly as long (if the project is going well) because the team should be in a groove and not drawing outside of the lines as much.

Conclusion
There are a bunch of other things that you can do, but this is a good punch list of things that should be set up from the beginning of the project.

Wednesday, December 3, 2008

Team Foundation Server - Pros and Cons

So you're shopping for a Source Control server. There are a lot of options out there. I've only used two systems in earnest: Source Safe (SS) and Team Foundation Server (TFS). For the most part, I don't hate too many products / technologies. I'd like to think that I'm a pretty open minded guy. However, I truly hate Source Safe. I despise it. I wish that it would go away and never bother anybody ever again. It is a horribly outdated piece of crap technology that should never see the light of day. If you're thinking about adopting Source Safe for a new project, don't. Just stop. Shoot yourself in the foot. It will safe you time and will be less painful in the end. Alan De Smit does a good job detailing why Source Safe sucks in great detail here.

There are a number of other choices out there. I've been particularly interested in trying SourceGear Vault. I've only gotten as far as downloading the trial - I never got around to installing it. I've used their Source Off Site (SOS) product (it's a client / server piece which provides much faster WAN / VPN access to a Source Safe database - it was absolutely brilliant).

But, back to TFS. Here's some background information that should help you get a feel for what you're getting into...

Pros:
  • It's fast as hell, at least compared to Source Safe. It's designed to run over a WAN and it communicates with the server via web services. Getting latest is very efficient because TFS only gives you what you need and doesn't need to check every file in your solution.
  • Integrates ever so nicely with Visual Studio. It fits like a Glove into Visual Studio 2005 / 2008.
  • Atomic Commits. In other words, if one file doesn't make it into a checkin, none of them make it. That's the way it should be.
  • Comes with Team Build, which provides functionality that I hear is relatively similar to CruiseControl.net (or so I've heard).
  • The work item subsystem is pretty sweet. It has lots of features out of the box and is very extensible. This is where you can store bugs, work items, etc. It's not as full featured as Clearcase or one of those systems, but it's adequate for doing most items.
  • Pretty solid extensibility. You can add your own check in policies, access the work item / source control via API (either via the client API or through the service interface).
Cons
  • Costs an arm and a leg. You can get a 5 person workgroup edition (for free with your MSDN subscription), but you'll have to saw off your arm (and leg) to buy enough seats for 6 people.
  • Setup is a pain in the ass. There are a bazillion (well, not that many) prerequisites and it takes forever to get them all installed (SharePoint, this, that, etc). Setting up the data tier on another machine takes some work as well. It's definitely not a turnkey, just run Setup.exe and accept the defaults installation.
  • Because installation is such a pain, recovery is a pain unless you backup the TFS as an image. The good thing is that you only need two things in a worst case situation to restore a TFS server: The SQL Server database backup and the report private key. Well, you can probably live without the key, but the instructions say that you need it.
  • The work item system in its current implementation does not support hierarchical data. That (very important) feature is supposed to be supported in TFS 2010. IMHO, you need a hierarchy to do a proper breakdown of any non-trivial project. I actually moved all the breakdown information into MS Project (shudder) to get a better breakdown.
  • No keyword expansion. This ruffled more than a few feathers as evidenced here. I've never used the feature in Source Safe, but to those who have used it, it is a sorely missed feature.
  • No sharing of files. This one is a bit annoying. While I've only used sharing sparingly, it's a very handy feature when you need it.
  • No shadow folder support. Once again, I've never used this feature. However, if you've depended on it - you'll miss it.
There are some notable differences for people that are used to Source Safe, such as:
  • TFS keeps track of everything in a Workspace. You have to get all the files that you want to work on before you can do anything. Also, if you don't check out a file in TFS, but you just make it writable, it will likely get overwritten because TFS didn't know that you wanted to do that. This causes all sorts of consternation for long time users of Source Safe. Martin Woodward does a good job of explaining Workspaces here.
  • Compares work a bit differently. In Source Safe, if you do a compare of a directory, SS will literally compare every file byte for byte and report any differences (which could take an eternity). On the other hand, TFS assumes that if you haven't checked out a particular file, it isn't different. That's how TFS operates so quickly. Most of the time it works the way you want, but once again, it's different than how it worked in SS.
I'm sure that there are many more pros, cons and differences, but those are the big ones IMHO / experience. Whatever Source Control solution you end up picking, make sure that you trial it to make sure that you can live with it. Once you have a bunch of data in the source control database, it's usually non-trivial to move it to another server.

ProjectGuids in Project Reference


Why is it that it's always the little details that take the most time in any project / implementation? In my recent efforts to get Team Build up and running, I ran into some of the oddest things. The one that had me pulling what's left of my hair out was a very small thing - the .

Normally, in the course of developing a .NET project, you can pretty much ignore the little details like what the projectGuid is for a given project. The problem that I ran into is that one of the project references had an oudated value.

This is the little beasty that was causing all of the issues (this is in the raw xml of the .vbproj file):
<projectReference include="..\MyClassLibrary.vbproj">&
<name>MyClassLibrary</name
>
<project>{D5D5C0F8-6BEB-4415-9CA0-E2A84B8D32B6}

<package>{F184B08F-C81C-45F6-A57F-5ABD9991F28F}</package>
</projectReference>
It's rare that a projectGuid will ever change. In this case, we had to make the MyClassLibrary a VSTO plug in for word. I didn't do the conversion (and wouldn't have caught something like the projectGuid changing if I did), but for some bizarre reason we ended up with a new projectGuid for MyClassLibrary.

This is the little bugger that changed in the MyClassLibrary.vbproj file:
{D5D5C0F8-6BEB-4415-9CA0-E2A84B8D32B6}
The fix was simple, all I had to do was copy the correct Guid to the tag of the other project and I was good to go. Finding that was absolutely horrible though.

Along the path, I found out that the solution would actually build after a complete rebuild. Apparently there is some voodoo happening with the ResolveAssemblyReference.cache files. I'm guessing that these "fix up" the references without changing anything in the original .vbproj files. We didn't notice it in our local build environment because these files were already generated with the correct overriding information. However, the build machine always grabs the source to a clean directory, so these files haven't been generated yet.

Tuesday, December 2, 2008

TFS Team Build

It was the best of times, it was the worst of times. Team Build (as included in Team Foundation Server 2008) is a very powerful tool which can save all sorts of time by automatically building your code every time somebody checks something in. I've never used it, but TFS is supposed to do a lot of the things that Cruise Control.NET does out of the box. That said, it can be a pain tracking down all the resources you'll need to get your solution compiling like a champ. Here's what I did / used.

For the most part, getting started is pretty straightforward. However, there are a few gotchas. In general, you want a dedicated server to do the team builds. To get started with that, you need to install the build agent on that machine. The installation bits are on the Team Foundation Server install disc. The important part here is to set up the build service to run under an account that has access to TFS. If you screw this up the first time, just go into services and change the username / credentials for the "Visual Studio Team Foundation Build" (don't forget to restart the service).

After you've installed that, you need to set up your workspace. I followed Buck Hodge's advice on getting that going.

Now, some special cases. If you are signing any of your assemblies with a .pfx key, you'll want to import that onto the local machine. Here's some advice.

Another item to keep in mind is that the solution will be built with a slightly different folder structure. This can cause problems if you have a post build step that copies files (e.g. if you have some plug in type files that aren't directly referenced by the main application project but the app needs them to run).

This approach work fine on the local machine, but not on the build server. To make something like this:
copy "$(TargetPath)" "$(SolutionDir)MyApp\bin\Debug"
work on the build machine, you can do something like this:
IF EXIST "$(SolutionDir)MyApp\bin\Debug" copy "$(TargetPath)" "$(SolutionDir)MyApp\bin\Debug"
That should work in most cases (your mileage may vary).

I haven't tried setting up an unit tests to run automatically. If / when I'll do, I'll likely post about that too.

Monday, December 1, 2008

Internationalization


How many times have you gone through this scenario? You start out a project, and you ask all the right questions including:
Will this application ever need to support multiple languages?
More often than not, this answer will be "No, that's crazy. Don't add support for that. It will take way too much time".

Now fast forward months / years into that project. How many times do you find out (usually in a conference call) that some huge customer in Elbonia that wants to buy a bazillion licenses of your software, but only if it's localized in Swahili.

This is the part of the story where the developers usually try to explain how difficult such an endeavor would be and that the next planned release will be postponed by orders of magnitude.

As in most cases, an ounce of prevention is worth more than a pound of cure. The only way to avoid this miserable situation is to build in internationalization support from the beginning. There are numerous objections to supporting this from the beginning:
  • It will take too much time to add support for multiple languages
  • It's too burdensome to add new strings
  • Working with the UI is too much of a pain
These concerns can't be dismissed. These are valid complaints / obstacles. However, it's worth pointing out that there are some benefits to going this route from the beginning, even if the PHB never makes the dreaded "We need to sell this thing in Russian" call:
  • All of the strings are in one place, so it's easy to maintain phrasing style for the whole project even with multiple developers.
  • Common strings can be reused (e.g. Cancel, OK, Username, etc). If a decision is made to change one of those, it gets changed everywhere.
  • Testers can usually make changes to spelling, phrasing, etc without bothering the developers. I've spent way too much time correcting spelling, phrasing errors when a tester could have just done it themselves if they had ready access to the strings.
Oh, and when you get the last minute call that you need to translate the product, it will be no big deal.