Documentation: It Should Be A Priority

frederick

Well-Known Member
Reaction score
154
Location
Phoenix, AZ
Too many times have I seen a beautiful network setup and no one knows how its been setup. When I've asked about certain specifics concerning the network or systems, I'm looked at with a deer in the headlights look. Nothing to outline common and uncommon tasks, no change logs or management, or what the configuration of a system or network is.


Documentation is extremely important to any company, technician, and sometimes a user, who are going to be maintaining or using the network and systems. Without proper documentation, when problems occur, there is no reference to go off. Technicians and administrators don't have a master document that states what the network is supposed to look like, there is no history of changes, or problems, or even maintenance tasks.


My most recent experience about the importance of documentation comes from a job interview with a large international corporation. During the interview, I was asked if I knew how to recover the Root password on a Unix system. I honestly admitted to them that I had not done that since a school setting in 2007, and that I do not remember how to perform such an action. The interviewers response was shocking to me. I was informed that they have to perform this tasks at least once a year at their various sites because administrators would change the password as per their SOP every 3 months, and that when the administrator would leave or be transferred no one would know how to access to the server if it needed to be rebooted or if a certain task needed to be performed.


I stopped the interview right there, and began asking them questions about their documentation management and other processes that they had in place. They went on the defense, saying that they have these documents and processes in place, but based on their other responses I would have to say I differ. I told them thank you for the opportunity and left.


For every minute a critical system is down, a business is losing money. While many problems with servers and network devices may be able to be corrected without referring to any documentation, technicians and administrators can use documentation to track trends in maintenance, usage, problem resolutions, and even what the last technician or administrator did on that system.


My personal philosophy on documentation uses a color coded system to document everything and separate various information based on where it belongs.


Purple Book/Smart Book: Also known as the “Smart Book”, this is a series of documentations that are specific to a single system. Usually in both a digital and hard copy, this book would include vital information such as passwords for root and critical system user accounts, configurations, baseline reports, common and uncommon maintenance tasks, and specific troubleshooting tasks. Each system needs to have its own Purple Book. In the hard copy needs to be a notebook to log changes, when tasks have been performed, problems resolved, and anything else that might need to be added to the book that affects or tracks anything having to do with the specific system.


Another component of this book is the Mean Time Before Failure/Replacement (MTBF/R) table that outlines when specific components needs to be replaced, or when the whole system needs to be replaced. As well as an inventory of dedicated on-hand replacement parts that are specific to the system.


Red Book/Infrastructure Management: When discussing the backbone of any network, having a reference to look at for both the physical and logical layout of the network helps to make plans and changes to the network. This book will cover every router, switch, wireless access point, and network cable connections. As well as documenting where all the computers, servers, printers, and phones are located in all of this (physically and logically). A detailed outline of the security posture of the network is also present in this book. Maintenance tasks, baseline reports, configurations, troubleshooting and maintenance tasks, and anything else needing to be documented is all contained in this book.


Like the Purple Book, the hard copy needs to have a dedicated notebook outlining the same information as that found in the Purple Book. As well a MTBF/R table needs to be in the book.


Blue Book: Usually only a digital copy, this is a consolidation of all the maintenance tasks needed be performed across the network and systems. How this book is put together is going to vary based on how the IT Manager wants to organize everything, but it should be easy to navigate by all technicians and administrators without much training.


Daily, Monthly, Quarterly, and Annual Maintenance tasks need to be placed in this book. The way I have always organized this book is the first section covers all daily tasks. The second section covers all general monthly tasks that need to be performed. The third, fourth, fifth and sixth sections cover quarterly tasks in the order of Q1, Q2, Q3, Q4, as not all systems may need quarterly maintenance. Annual tasks can all be added in a single section, or broken up throughout the quarterly tasks to stretch out the workload so not all maintenance tasks are performed at once.


This book should not include anything like specific network or system information. And should not have troubleshooting techniques either. This is simply a consolidation of maintenance tasks so that any one technician or administrator can use this digital book as a reference and have complete access to what common maintenance tasks need to be performed throughout the network.


Green Book/Change Log: This is for the management if you will to see what is going on with their IT department. I've used this book as a consolidation of any changes that have occurred that are “public” access, and when giving briefings on what has occurred over the last X amount of days to make my point in persuading management to make a move on certain purchases, changes, or to inform them of certain situations.



Assembling these various books, and maintaining them, requires that everyone from the top to the bottom enforces the usage of these books. Creating or finding a database system can assist greatly in accomplishing a proper documentation standard. There are also several programs that can help you put together all your documents, and many you probably already have installed on your computer.


Any Word processor can be used to assemble task templates, configuration settings, etc.

Any Spreadsheet or Excel program can be used to put together any tables that are needed easily and fairly quickly.

Any Drawing or PowerPoint program can be used to put together your maps showing the interconnections of devices, printers, servers, and network equipment.


Other programs such as Microsoft Visio, Open-AudIT, Network Notepad, and others have all been designed to help assemble critical components to proper documentation.


Within the department, at least one person needs to be responsible for the documents, to ensure they are assembled and updated correctly. This doesn't mean one individual does all the work, because it is important that every member of the IT department adds their changes to the various books, and keeps them updated. But at least one person needs to ensure that no duplicate, or outdated information remains, and that any official changes are accurate and appropriate.


If proper documentation is put in to place, and management enforces the maintenance of a proper documentation process, if new employees are brought in, or if a technician is brought in to finish what another started, then these books and documentations can be used to ensure that all personnel are on the same page. Proper documentation also helps to enforce department standards, create training outlines, and stay on top of important management tasks such as capacity planning and resource management. Without proper documentation, time can be wasted on technicians and administrators trying to relearn uncommon tasks, figuring out current configurations, and researching problem resolutions that have occurred in the past from outside resources.


It should be every companies, and every IT departments priority to implement a proper IT documentation program.
 
I like how you not only color code things, but actually have them on paper. People say "It's in the cloud and on the file server. If it crashes, we still have a copy." I always like to follow that with "If the core router/switch dies and both are offline, can you manually configure a device to re-establish a connection?"

I am a big fan of pen and paper documentation. (I'll also never accidentally format it or have it hit with CryptoWall...).
 
This seems geared to in house IT departments. Most of us here are don't fit that description.

For those of us that perform these functions on a contract basis for multiple clients, documentation is still important. But creating documentation isn't free.

We try to document as much as we can of the networks and configurations for our clients. The question becomes, who does that documentation belong to?

My opinion is that the customer is entitled to a list of all passwords and account numbers for customer owned systems . Anything else, diagrams, notes, etc. belongs to us unless they pay to have it created. If we create it to do our job better, they are tools we use to do the job.
 
Last edited:
I like how you not only color code things, but actually have them on paper. People say "It's in the cloud and on the file server. If it crashes, we still have a copy." I always like to follow that with "If the core router/switch dies and both are offline, can you manually configure a device to re-establish a connection?"

I am a big fan of pen and paper documentation. (I'll also never accidentally format it or have it hit with CryptoWall...).

And if you spill a cup of coffee on it, it just turns brown.
 
"It should be every companies, and every IT departments priority to implement a proper IT documentation program."

Yes, very true, but in reality it won't happen. I've worked in many New York data centers in the 80's and 90's and for small IT shops in the early 2000's and documentation was always something everyone thinks they should do, but it just never gets done. Even today with small shops I deal with, almost nothing is documented. Documentation is the thing everyone wishes they had after a disaster, but they dont care about before. Even after the disaster people will talk about working fixing it but nothing happens.

It's like backups. Everyone agrees they should be done and yet they still don't do them. After a failure everyone says they are going to get tough on backups but then nothing happens.
 
Yes, very true, but in reality it won't happen. I've worked in many New York data centers in the 80's and 90's and for small IT shops in the early 2000's and documentation was always something everyone thinks they should do, but it just never gets done. Even today with small shops I deal with, almost nothing is documented. Documentation is the thing everyone wishes they had after a disaster, but they dont care about before. Even after the disaster people will talk about working fixing it but nothing happens.

It's like backups. Everyone agrees they should be done and yet they still don't do them. After a failure everyone says they are going to get tough on backups but then nothing happens.

Exactly!! Especially residential customers. They will almost never ever backup unless it's something totally automated, but if the automation fails then guess what...lost data lol. Even then though most don't seem to spring the extra buck for anything with backing up files.
 
We try to document as much as we can of the networks and configurations for our clients. The question becomes, who does that documentation belong to?
The documentation belongs to both. With PCH, we could easily create the foundations of these books through the RMM you use. Part of each purple book includes the breakdown of each system (RAM, HDD's, CPU, etc.). If you don't know install dates, go off the systems "install" date and work from there. With the Red Book, same thing with assets and inventory. This information belongs to the client. Not you. And through your RMM, it should be easy to export the data and put it in to a document format.

With the Tier 2 (Unlimited Remote + Proactive Maintenance services) and Tier 3 (AYCE), techs would be required to document the configurations of the systems and devices. A template was created, and easy to fill in via your laptop or tablet. Again, word document format. Some routers allow you to export the configuration settings, these log like files can be used in lieu of some forms. This information belongs to the client, but is maintained by you.

For the network map, various applications can do this for you. It's on you to figure out which one works the best and does what you need it to do. Owned by the client, maintained by you.

The green book is composed of your trouble tickets. Too easy to integrate in to something. These documents are shared ownership. Anything proprietary (scripts, programs, etc.) created or owned by you is your property, and is not available to the client to view. Things that you add to the trouble ticket labeled as "Customer Notes" are what the client sees. Hopefully you aren't putting in your trouble tickets "problem solved" and calling it a day. I outline what was the problem, what was done to correct, and what needs to be done to prevent it from happening again.

The blue book in a computer shop or MSP is 100% yours. I created documents for standard maintenance procedures in preventive and proactive tasks. This is how you create the standards for your techs to work and accomplish tasks.

My opinion is that the customer is entitled to a list of all passwords and account numbers for customer owned systems . Anything else, diagrams, notes, etc. belongs to us unless they pay to have it created. If we create it to do our job better, they are tools we use to do the job.
If a client hired me for a project, say a network infrastructure install, all diagrams, notes, etc., are the property of the client, but are created by me. One of the things that irked me was when a client would fire their previous MSP and then hire me...and the previous MSP took everything with them (passwords, diagrams, etc.). I would have nothing too go off for how the network is supposed to work, or why things are done in a certain fashion. Is it not the clients infrastructure? Did they not hire you to be their project manager? Many companies don't have full time project managers and teams, but instead outsource when projects come up. Also, why you may think it does nothing to help you in the long run to give the client the details on what you did and why you did it, it's always come back to me in a positive light. Especially if I'm called in to remedy something. With some clients where they had an internal IT department, handing over the necessary documentation is a lot easier than trying to teach them every single step and action taken in what I did.

documentation was always something everyone thinks they should do, but it just never gets done.
I personally feel that it is a failure on management in why this doesn't get done. Throughout my career, I always made it my mission to ensure that all the proper documentation is maintained, or delegated its completion. Last thing I needed was a tech calling me at 4AM about something that should have been put in the book.

Lets make sure we are all on the same page about this, for the most part, the Red, Purple and Blue books will not change very often, or even that much unless something seriously needs to be changed or added. The green book is the only one that is going to be doing all the changes, and this is to ensure you are tracking trends, and possibly scheduling for more serious work to be done. As well as keep the upper management/client informed. If you got the templates put together in a word document, all you have to do is fill in the blanks.
 
When it comes to critical passwords and such, I believe in what I was taught in the military.

Everything that requires a password or combination has an envelope with that information written on a card sealed inside it. On the outside is a date, what it's for, who created it and their signature.

All of these envelopes are then placed in a safe (usually a department head or such). When access is needed and there is no one around that know the password or such, they go to the safe and retrieve the correct envelope. Afterward the password or such is changed and a new envelope is made and placed back into the safe.

This solves the problem of someone leaving and no one knowing the passwords and such.

Having every password and such listed in a book means anyone looking in that book can see everything even the ones they should be seeing.
 
Problem is that most businesses, especially small business, are run far less strict than that. Passwords are setup to keep the outside world out of the network but everyone knows everyone elses passwords and they do not want that changed. They usually only learn this the hard way when one trusted employee abuses that knowledge.
 
This seems geared to in house IT departments. Most of us here are don't fit that description.

For those of us that perform these functions on a contract basis for multiple clients, documentation is still important. But creating documentation isn't free.

We try to document as much as we can of the networks and configurations for our clients. The question becomes, who does that documentation belong to?

My opinion is that the customer is entitled to a list of all passwords and account numbers for customer owned systems . Anything else, diagrams, notes, etc. belongs to us unless they pay to have it created. If we create it to do our job better, they are tools we use to do the job.

It's geared towards ANYone.
Say you're a 1x man show. You get hit by a bus! You're gone, and your clients have to go out and find some new nerd to take care of their systems. It's more beneficial to your client if that new nerd has a layout of their systems, the new guy can quickly start supporting your past clients better.

Also, as/if you grow as an IT support company, you have more staff. Pretty soon you're mixing up which engineer supports which client. We're at that point now....previously we all just supported our own clients, but we're at a growing point where we had to revamp our helpdesk system and start mixing it up. Cross training if you will. We're doing a similar thing within our HelpDeskManager...putting up notes under each clients FAQ section so we call can access notes for the client.
 
Why not bill the client for the documentation? I don't see why a client would refuse it, especially if you tell them they own the documentation and even offer an annual update to make sure everything stays current.

I don't care how small the client is (as long as it is a business not individual). I think it would be a great advantage over other techs who don't offer it (which is probably 99% in my area), in fact I think it is such a great idea I will start offering it. Heck if you really don't have the time get an intern or college student to write it for you. It doesn't have to be really complicated (as outlined in the original post) but let them have something outlining how their system is setup.
 
Last edited:
I document every business customer whether they pay to have it documented or not. The upside pays for itself when I am on vacation whoever is covering for me doesn't have to call and bother me to ask for the login to a client's router, etc. Although this doesn't help if the tech covering for you doesn't know how to read said documentation lol (honest to God that happened).
 
It's geared towards ANYone.
Say you're a 1x man show. You get hit by a bus! You're gone, and your clients have to go out and find some new nerd to take care of their systems. It's more beneficial to your client if that new nerd has a layout of their systems, the new guy can quickly start supporting your past clients better.
I have no problem documenting. Customers own the equipment, they are entitled to the passwords and access to their equipment. We also gather a list of their providers, along with account numbers and contact info, and give that to them. If some one needs to get into the system, they should have no problem. If they replace us, they don't have to contact us.
Also, as/if you grow as an IT support company, you have more staff. Pretty soon you're mixing up which engineer supports which client. We're at that point now....previously we all just supported our own clients, but we're at a growing point where we had to revamp our helpdesk system and start mixing it up. Cross training if you will. We're doing a similar thing within our HelpDeskManager...putting up notes under each clients FAQ section so we call can access notes for the client.
Do you provide these Notes and the FAQ to the client? Is it part of your MSP package? How much of your documented work do you pass on to the customer?
 
Everything that requires a password or combination has an envelope with that information written on a card sealed inside it. On the outside is a date, what it's for, who created it and their signature.

All of these envelopes are then placed in a safe (usually a department head or such). When access is needed and there is no one around that know the password or such, they go to the safe and retrieve the correct envelope. Afterward the password or such is changed and a new envelope is made and placed back into the safe.

This solves the problem of someone leaving and no one knowing the passwords and such.

Having every password and such listed in a book means anyone looking in that book can see everything even the ones they should be seeing.
I did the same thing in the Army, but with "civilian" clients like what you might deal with this might be a little overkill and chances are they would forget. These books are not laid out on some random desk.

Individual user accounts are not in any of these books. Only user accounts that matter to the system operating properly. Another thing you can do is list these system accounts and omit the passwords, and then store the passwords separately.

I've always placed the hard copies in the owners/managers office, with a title like "Red Book" or "Purple Book", etc, usually in a lockable cabinet. Digital copies are placed on wiki server requiring a username and password to view anything, and placed behind a SSL, port 80 disabled, and user account lockout settings.

Passwords are setup to keep the outside world out of the network but everyone knows everyone elses passwords and they do not want that changed.
Try educating your clients. The majority of the ones I maintained changed things after their education. If they are serious about their business, they'll change things. You'll most likely never get 100%, but as long as you work on it, many clients will hopefully change their ways.
 
A slightly off topic question...where do people store all this detailed documentation whilst also making it available to onsite techs?
 
Individual user accounts are not in any of these books. Only user accounts that matter to the system operating properly. Another thing you can do is list these system accounts and omit the passwords, and then store the passwords separately.

Digital copies are placed on wiki server requiring a username and password to view anything, and placed behind a SSL, port 80 disabled, and user account lockout settings.

The only thing that would be treated as I described would be passwords and such that are only known to a select few, like Admin passwords, Equipment Usernames and passwords (switches and such). Basically those that are critical like the Root password you describe in your OP.

So what happens to the digital copy when that manager leaves and no one knows the password to the wiki server? It's the same scenario as you described involving the Root password.
 
The only thing that would be treated as I described would be passwords and such that are only known to a select few, like Admin passwords, Equipment Usernames and passwords (switches and such). Basically those that are critical like the Root password you describe in your OP.

So what happens to the digital copy when that manager leaves and no one knows the password to the wiki server? It's the same scenario as you described involving the Root password.
In the case of the wiki site that was hosted by PCH that had our digital copies, that clients could access, their username was assigned by us. So if a manager left, the owner could call up amd say Tom no longer works here, and we could remove Tom from the system and add the new manager.

Technicians should have some administrative priviliges with their own admin account.
 
A slightly off topic question...where do people store all this detailed documentation whilst also making it available to onsite techs?
My employer uses PC Repair Tracker (aka PCRT). It is web based, (I think it's only HTTP though -- unless my boss opted for no encryption) so anyone with an account can log in and view the information.
 
A slightly off topic question...where do people store all this detailed documentation whilst also making it available to onsite techs?

We use our RMM/HelpDesk tools. In N-Central, under each SO, under each node...such as a server or switch or firewall, there is a "Notes" section there. A bit over a year ago after Solarwinds bought up N-Able, Solarwinds integrated their Help Desk Manager...and we're now shifting over our notes to that....under a FAQ section under each client.
 
Back
Top