The Joel Test: 12 Steps To Better IT Management
Sep 14, 2009 at 3:59PM Thomas Ptacek (With apologies to Joel Spolsky, from whom this post was ripped off)
Have you ever heard of COBIT? It’s a fairly esoteric system for measuring how good a network security team is. No, wait! Don’t follow that link! It will take you about $3,500,000 just to understand that stuff. So I’ve stolen my own, highly irresponsible, sloppy test to rate the quality of a network security team. The great part about it is that it takes about 3 minutes. With all the time you save, you can go to law school.
(I’m using network security as a synecdoche for all of enterprise IT, but I think I’d win an argument about whether the same issues apply to the team that manages the WebSphere deployments and all the WAR files and whatnot.)
The Joel Test
- Do you use source control?
- Can you make a build in one step?
- Do you make daily builds?
- Do you have a bug database?
- Do you fix bugs before writing new code?
- Do you have an up-to-date schedule?
- Do you have a spec?
- Do programmers have quiet working conditions?
- Do you use the best tools money can buy?
- Do you have testers?
- Do new candidates write code during their interview?
- Do you do hallway usability testing?
1. Do you use source control?
Programmers have had source control since the 1980s. With source control, every change you make to a file is tracked. Any line of code can be tracked diff-by-diff back to the origin of the file. Are your firewall rules under version control? If you don’t have source control, you’re going to stress out trying to get engineers to work together. They’ll have no easy way to know what their coworkers did. Mistakes can’t be rolled back easily. And source control backs up all your rules, in a single place, so you can reconsitute that PIX ASA that just threw a rod from a cold spare in minutes.
2. Can you make a build in one step?
By this I mean, how many steps does it take to get an access rule change deployed from the latest source snapshot? On good teams, there’s a simple process you can follow to get a firewall completely deployed from scratch, which checks out the company standard configuration, adds the right doodads to the configuration, tracks the device in inventory, etc.
A process that takes more than one step is prone to errors. And when you’re under pressure because Pepsi needs you to punch a hole for a database management app that Coke has forbidden you from allowing near their dat,a, you want to have a very fast cycle of making sure your rules work. If it takes 20 steps to deploy a rule, you’re going to crazy and you’re going to bring the network down.
3. Do you make daily builds?
When devs use source control, sometimes people check things in that break the build. The usual pattern is, things work fine on the developer’s machine, but she forgot to add a header file, so nobody else can build. “Breaking the build” can cost developers whole days of productivity, so good teams have rules to detect and punish people who do it.
Network security engineers have a “build”, but when they break it, they don’t just kill the rest of the team’s productivity. They kill the network. So good network security teams want to make sure people can work on projects that touch access rules without getting unreviewed changes into the configurations that will get pushed out during the next change window.
4. Do you have a bug database?
I don’t care what you say. If you’re managing access rules, even if you only have a few devices, without an organized database listing all the change requests that produced the rules, you’re going to have crappy firewall rulesets. Lots of engineers think they can hold the rules in their heads. Uh huh. What hosts on your network are allowed to talk FTP to the outside world, and why? Thought so. You absolutely have to track change requests formally.
Change tracking can be complicated or simple. A minimal tracking system needs to record the following facts for every request:
Who requested the change
Why they requested it
Was the request approved
Who was the request assigned to
What devices had to change to accomodate the request
You can absolutely DIY this; lots of teams build their own tracking systems in-house, and they work great.
5. Do you fix bugs before writing new code?
When Joel Spolsky worked on the Excel team at Microsoft, he picked up a story about Word for Windows, which slipped constantly because the schedule allowed no time to fix bugs. The quality of the codebase decayed, to the point where developers were writing “if 2+2 !+ 4 return 4” or whatever. They referred to this as “infinite defects methodology”, converted to “zero defects methodology”, which meant they got to fix bugs, and eventually shipped. And when Joel Spolsky was in the alps fighting grizzly bears, he used his magical fire breath and saved the maidens fair.
But I digress.
Once every year or so, big companies commission small companies like ours to do the “annual external pen-test”, in which testers try to break in through the perimeter firewall. Even though I don’t do a lot of network pen-testing, I’ve done a couple. And on all of them, some stale old Win2k host gets left exposed or some branch network has 445/tcp open, because there are 20,000+ lines of firewall rules and rules only get added, never removed.
Just like with code, it is much more expensive to fix a bug early than late. But at least with software, you find out about bugs because your program crashes or a user sees the wrong header font size in the help file. With firewall rules, not so much. You mostly find out that you’re boned when you flunk some random audit.
Most of the same reasons developers need to fix bugs before writing new code apply to enterprise IT, too:
It’s easier to fix a bug when it’s right there in your face than to remember it or track it down later
It’s easier to predict to your customers when their changes are going to get completed when you know you aren’t going to lose 2 days fishing old SMB exceptions out of your rules after an MSRC announcement
It’s less stressful and less costly to fix things up front than to blow a change window or push an emergency change because of an advisory or a client audit.
6. Do you have an up-to-date schedule?
Which brings us to schedules. Because the dirty secret is, the rest of the IT team at your company probably hates you, because your job mostly involves saying “not yet” to their requests to change things on your devices. But that doesn’t cut it. Too many business processes are impacted by your workflow; clients can’t get brought online until the PMP-certified project manager checks off your box in the process, and so you’re a cost center, and the VP/Operations starts to hear about how one of next year’s priorities is to “streamline firewall management and reduce TCO”.
It’s possible to keep a schedule; figure out how fast you can complete a change, and track the number of outstanding change requests you have. It can take significant time to research and execute a complicated firewall architecture change, as long as you can communicate up-front to project teams how long it will take, and then come in on schedule.
7. Do you have a spec?
Documenting all your configuration is like writing a software spec: everybody agrees it’s a good thing, but nobody does it.
It’s weird that this is the case, because nobody in the company has a stronger opinion about how technology should be deployed and managed than the network security engineering team, but probably all you have is apocryphal Visio diagrams on a network share somewhere, and you’re much more likely to just throw another configuration line onto a device than to write a document that explains what you’re doing.
Documentation doesn’t have to be painful. You can just set up Mediawiki and start by writing a short paragraph for every device you manage. Or you can write programs that take inventory and analyze your configs. However you do it, you should be able to have a rule that says “no changes without updating the documentation”.
8. Do programmers have quiet working conditions?
And also, pet unicorns?
9. Do you use the best tools money can buy?
“Top notch development teams don’t torture their programmers […] and programmers are easily bribed by giving them the coolest, latest stuff”.
Things it’s crazy network security engineers don’t get to have include:
A network switch on their desk
Two monitors
A laptop with a big screen
Unlimited disk space
A place to get a new VMware image up with a couple of clicks
A license for Burp Suite
I could go on and on here, but the only reason I’m writing this is that Joel Spolsky has this as a Joel Test item. Motherhood, meet apple pie, and also see “pet unicorns”.
10. Do you have testers?
“Skimping on testers is such an outrageous false economy that I’m simply blown away that more people don’t recognize it”, which is probably why most network security teams have testers. There had to be something network engineering teams do better, processwise, than developers.
11. Do new candidates write code during their interview?
It’s hard to write code in an interview. Interviewing teams are so infamous for skipping this step that there’s a whole interview question methodology, the “FizzBuzz test”, that says “at least show me you can print the numbers 1-100 with every 3rd number as ‘fizz’ and every 5th as ‘buzz’”, which is a 1-liner in Ruby, but that’s how desperately teams need to know that candidates even know where the parens and the braces go.
And so here’s another thing network security teams do better. Because, do security engineering teams have this problem? Probably very few candidates actually know how TCP flow control works, or whether they actually need path MTU discovery enabled, but I tend to doubt that a lot of teams are staffed with people who couldn’t punch an exception into an IOS ACL set for FTP or DNS if they needed to.
12. Do you do hallway usability testing?
And here I concede that there is one item on the Joel Test that does not directly apply to network security teams.
My Point, And I Do Have One
There’s a seed funding firm called Y Combinator that is all the rage with the kids these days; they’ll give you ~$20,000 for your 2-person startup in exchange for 5-6% of the company even if you have almost no working code and you’re just out of school. Some surprisingly good companies have come out of YC. One of them is Dropbox, a hugely popular and powerful file synchronization system. A few days ago, Dropbox’s application to YC was posted. One of the grafs in the application talked about Subversion, and ended with “hackers [(developers)] have access to these tools, but normal people don’t”.
And its true of network engineering teams too. In a lot of shops, Subversion is space alien technology (and yes, it’s true of some dev shops, too). And where problems are recognized, as with issue tracking, they’re “solved” by massive enterprise management systems with their own headcount dedicated just to keeping Remedy working properly, and everyone hates it.
And of course this is a self-serving post, because “solving the Joel Test for firewall admins” could be the thesis statement for our product. But then, I’ve recently changed up roles at Matasano, and am managing Playbook instead of consulting, and I want to start talking more about what we’re doing. And here’s a way to start the conversation.
Good to be talking to you all again, by the way.
Reader Comments (70)
Really good resource, simple, but definitely what big consulting companies charge a few $10ks or $100ks to setup. Nice post.
COBIT is a consideration for many organizations because it's the primary IT controls framework utilized by SOX auditors (external or internal). However, I have to admit that a principles based framework is cheaper and easier to implement, or perhaps a hybrid framework such as COSO.
While controls-heavy frameworks like COBIT make little to no sense for an organization who doesn't need it (or that has scoped certain parts of their network out of regulatory requirements through proper segmentation, including IT/Ops monitoring), there are interesting frameworks such as Visible Ops Security that are easier to digest and include much more relevant information than "The Joel Test".
Some of your suggestions, such as the Burp Suite one, are gratuitous/unjustified. When verification web applications to achievable risk standards, it's best to utilize OWASP ASVS L2B, L3, or L4. Of course, an OWASP ASVS L2A can be performed with something like Burp Suite for a majority of situations (note: not all situations) -- but I think there is less value from an L2A approach in comparison to an L2B approach. Also, L2A verifications will likely take more time than an L2B verification, unless somehow the source code is unusually larger and more complicated than the runtime app (note: I'm not sure I've run into this situation, but I'm willing to give in to the possibility).
I'd also like to make a few comments on "A place to get a new VMware image up with a couple of clicks". First of all, I think that most organizations would benefit by using a neutral format for images, such as OVF. Secondly, "getting" images up is less useful than cloning images or loading them from templates. Lastly, I'd like to suggest that Microsoft's free Hyper-V Server 2008 R2 with the free Citrix Essentials for Hyper-V is a much better option for organizations because it's powerful (i.e. has cloning/templating) and is free (albeit not open-source). VMware has no similar offering, nor does Linux Xen or Linux KVM (although it is possible that these may be capable in the future). So, in effect, I'd rather hear of VHD format images (Microsoft proprietary) instead of vmdk (VMware proprietary), although I do see that I'm contradicting myself by saying that. Maybe a system that dumps VHDs in-use into OVF format every month on backup/snapshot?
Since I'm in an argumentative mood, I'd also argue that there is no benefit to punishing people who break things -- but instead to reward those who maintain or improve on stellar performance. I've seen this done well with Six Sigma or similar goal mapping using Business Scorecards, Key Performance Indicators, and SMART objectives. In "build terminology", I'd rather see something like PMD or StyleCop prevent check-in of code, instead of something like FIndBugs or FxCop "breaking builds".
Great job relating software to networks. As someone who does both, it's important to realize that a project is a project regardless of the skillset being used and good management practices are always important. I've been a big fan of the Joel Test as a basic starting point for getting a team functioning for a long time, but, I have to wonder what would Brian Boitano do?
Juicy clothes
Buy full line Juicy couture products from our site at a low price to make yourself a fashionista! Dress juicy couture clothing, holding juicy couture
Shop prom dresses, formal dresses, prom shoes, 2010 designer prom gowns at dres4sale.
for cocktail dresses, dresses for prom, homecoming dresses, and evening dresses. Cheap prom dresses or couture designer evening gowns for your next formal.
evening dresses
Evening Dresses. Women's Formal & Special Occasion Dresses ... Welcome to Cheap Evening Dresses for Sale! ... Buy Cheap Evening Dresses Sales & Accessories
prom dresses
IT management is so important for a company. I think this is the most important part of a company. I lot of money needs to be invested here to succeed.
whistleblower lawyers california
P90x .It really is not expensive if you factor in the cost
of a gym membership,P90x workout . The cost for P90X is
about three months of a paid gym membership but you get to
keep the program foreverP90x . You can try many of the
online sites, but it will be the same as buying from the
company or a Beachbody Coach. Make sure you are getting
original DVD's. People are selling copies all over. The
problem is how long will they last, P90x workout ,and you
truly need the exercise and nutrition guide to even follow
the program. You can go to any site
http://www.p90xmall.com/ or you can go to and click on
products. P90x dvd You can order directly from the
site,P90x dvd.
web hosting
email hosting
best hosting
Seo Services
SEO
search marketing
tiffany1837jewelry
tiffany1837co
tiffany jewellery
tiffany1837online
t925silvertiffany
iloveyou925
The article written by your very good, I like it very much. I will keep your new article.
rosetta stone spanish,
rosetta stone,
rosetta stone language.
These steps to a better IT management are definitely worth considering. It is a great way to determine and access the capability and ability of your team. Using the best tools that money can buy can be really hard especially to those who are just starting out, but it is definitely a great factor in IT success.
Duncan Samuel
Online Scheduling
Creates concern over uk Tiffany
role with NGOs, their goals and agendas. Should Tiffany rings
advocate NGO goals? What power should Tiffany earrings
have to tell NGOs what agendas to seek in exchange for the company's cooperation? Focuses attention on Tiffany necklaces
, running the risk that the company will "stand alone" among its competitors.
Great! Do you know the MBT Beach, MBT Changa, MBT Tembea and MBT Tunisha? They are good shoes. I am sure you will like one of them. They are for our smart young ladies. If you want to buy them, please contact us as soon as possible. We will give you some discount.
embroidery digitizing | custom digitizing | embroidery punching | english to urdu translation | business card printing | panaflex printing | stationery printing | seo training in karachi | seo services | seo expert | printing services pakistan | embroidery digitizers
Tiffany jewellery
place in America's popular imagination was given a boost by Truman Capote's 1958 novel Breakfast at Tiffany rings
and the 1961 film of the same title, in which Audrey Hepburn as Holly Golightly accepts from her admirer a silver rings
toy Crackerjack ring. Michael J. Kowalski, chairman and CEO of Tiffany earrings
., the world-famous luxury jeweler and specialty retailer.
The overdriving of writers from traditional antiquity and the manner pursuit is unswerving and eternal topic of vuitton replica . Moreover large much become of other provide Handtassen with varnish with louis replica , grants, portfolios supply several styles in this online shop to cheap sale prices. The can splendid and abundance search of the Handtassen of louis vuitton replica , portfolios and grants, select your own idealized boulevard M93642 of varnish Wilshire of the monogram INCREASED of this fabulous sale rise.
This large format louis vuitton mens wallet is an outstanding piece among all the gorgeous fashion designer handbags, due to its sophisticated design and fabulous details. As you may have noticed, this discount louis vuitton bags takes its name from the city of Beverly Hills, which is almost the synonym of celebrity and top fashion. It is the same with this replica louis vuitton handbags
Finding the right gift for yourself ,your friends and your relatives in juicy couture,of course ed hardy t shirts and juicy couture jewelry also you good choise.juicy couture jewelry,juicy couture sunglasses,juicy couture handbags,juicy couture charms,juicy couture t shirts,juicy couture bikini,juicy couture tracksuits must be lovely and fun to look at.Juicy couture has over 100 different items available to choose ,such as juicy couture necklaces,juicy couture earrings,juicy couture bracelets,juicy couture wallets,juicy couture swimwear,juicy couture watches,juicy couture sunglasses,juicy couture socks,juicy couture flip flops,Juicy couture sale,Juicy couture sale,juicy couture panties.Birthdays are always worth celebration. Everyone wants to celebrate his/her birthday with others, and you cannot imagine a birthday celebration without juicy couture.
Forces of Evil struck again, 70-680 providing us with Ubuntu Satanic Edition - Fiery Spawn. USE is a 220-701 set of four moody themes (screenshots) tailored for those grown up kids who like to MCITP Certification harm small animals and steal holy water from local churches. 70-620 Exam
Safe, Faste, billig, wow cd key
You write good articles, I will always be concerned about
nfl jerseys
cheap ladies handbags purses online
cheap bags
cheap wedding dresses discount
tea length wedding dresses simple
cheap plus size wedding dresses
very good!
ed hardy hoodies
cheap christian audigier bikini
ed hardy shirts
cheap christian audigier hoody
women ed hardy clothing
ed hardy hoody uk
ed hardy jacket
ed hardy uk
ed hardy shirts
ed hardy swimwear
ed hardy ugg boots
ed hardy shoes uk
ed hardy hoody uk
buy ed hardy shirts
ed hardy womens shoes
ed hardy handbags uk
ed hardy at uk
ed hardy t shirts uk
ed hardy sunglasses uk
ed hardy womens tiger hoodie
Surprise! you will get the bag!
ed hardy clothing
ed hardy clothing
ed hardy shirts
louis vuitton
louis vuitton blog
louis vuitton
christian louboutin
Rosetta stone
louis vuitton
christian louboutin
Rosetta stone