Though I have fun working on SynthNet and other projects at night, during the day I fill the role of mild-mannered network administrator at the Manchester-Boston Regional Airport (actually, the day job is quite a bit of fun as well). One of the ongoing projects I’ve taken on is adding all of our various Intranet-oriented services into a single platform for central management, easier use, and cost effectiveness. As mentioned in a previous article (linked to below, see NMS Integration), I knew Drupal was the right candidate for the job, simply due to the sheer number of modules available for a wide array of functionality, paired with constant patching and updates from the open source community.  We needed a versatile, sustainable solution that was completely customizable but wasn’t going to break the bank.

The Mission

The goal of our Drupal Intranet site was to provide the following functionality:

1. PDF Document Management System
1. Categorization, customized security, OCR
2. Desktop integrated uploads
3. Integration with asset management system
2. Asset Management System
1. Inventory database
2. Barcode tracking
3. Integration with our NMS (Zenoss)
4. Integration with Document Management System (connect item with procurement documents such as invoices and purchase orders)
5. Automated scanning/entry of values for computer-type assets (CPU/Memory/HD Size/MAC Address/etc)
6. Physical network information (For network devices, switch and port device is connected to)
7. For network switches, automated configuration backups
3. Article Knowledgebase (categorization, customized security)
4. Help Desk (ticketing, email integration, due dates, ownership, etc)
5. Public Address System integration (Allow listening to PA System)
6. Active Directory Integration (Users, groups, and security controlled from Windows AD)
7. Other non-exciting generic databases (phone directories, etc)

Implementation

Amazingly enough, the core abilities of Drupal covered the vast majority of the required functionality out of the box.  By making use of custom content types with CCK fields, Taxonomy, Views, and Panels, the typical database functionality (entry, summary table listings, sorting, searching, filtering, etc) of the above items was reproduced easily.  However, specialized modules and custom coding was necessary for the following parts:

1. Customized Security – Security was achieved for the most part via Taxonomy Access Control and Content Access.  TAC allowed us to control access to content based on user roles and categorization of said content (e.g. a user who was a member of the “executive staff” role would have access to documents with a specific taxonomy field set to “sensitive information”, whereas other users would not).  Additionally, Content Access allows you to further refine access down to the specific node level, so each document can have individual security assigned to it.
2. OCR – This was the one of the few areas we chose to delve into a commercial product.  While there are some open source solutions out there, some of the commercial engines are still considerably more accurate, including the one we choose, ABBYY.  They make a Linux version of the software that can be driven via the shell.  With a little custom coding, we have the ABBYY software running on each PDF upload, turning it into an indexed PDF.  A preview of the document is shown in flash format by first creating a swf version (using pdf2swf), then using FlexPaper/SWF Tools.
3. Linking Documents – This was performed with node references and the Node Reference Explorer module, allowing a user friendly popup dialogs to choose the content to link to.
4. Desktop Integration – Instead of going through the full steps of creating a new node each time, choosing a file to upload, filling in fields, etc, we wanted the user to be able to right click a PDF file on their desktop, and select “Send To -> Document Archive” from Windows.  For this, we did end up doing a custom .NET application that established an HTTP connection to the Drupal site and POSTed the files to it.  Design of this application is an article in itself (maybe soon!).
5. Barcoding – This was the last place we used a commercial product simply due to the close integration with our barcode printers (Zebra) – we wanted to stick with the ZebraDesigner product.  However, one of the options in the product is to accept the ID of the barcode from an outside source (text/xml/etc), so this was simply a matter of having Drupal put the appropriate ID of the current hardware item into a file and automating ZebraDesigner to open and print it.
6. NMS (Zenoss) Integration – The article of how we accomplished this can be found here.
7. Automated Switch Configuration Backups and Network Tracking – This just took a little custom coding and was not as difficult as it might seem.  Once all our network switches were entered into the asset management system and we had each IP address, during the Drupal cron hook, we had the module cURL the config via the web interface of the switch by feeding it a SHOW STARTUP-CONFIG command (e.g. http://IP/level/15/exec/-/show/startup-config/CR) – which was saved and attached to the node.  Additionally, we grabbed the MAC database off each switch (SHOW MAC-ADDRESS-TABLE), and parsed that, comparing the MAC addresses on each asset to each switch port, and recording the switch/port location into each asset.  We could now see where each device on the network was connected.  A more detailed description of the exact process used for this may also be a future article.
8. Help Desk – While this could have been accomplished with a custom content type and views, we chose to make use of the Support Ticketing Module, as it had some added benefits (graphs, email integration, etc)
9. Public Address System – Our PA system can generate ICECast streams of its audio.  We picked these up using the FFMp3 flash MP3 Live Stream Player.
10. Automated Gathering of Hardware Info – For this, we made use of a free product called WinAudit loaded into the AD login scripts.  WinAudit will take a full accounting of pretty much everything on a computer (hardware, software, licenses, etc) and dump them to a csv/xml file.  We have all our AD machines taking audit during logins, then dumping these files to a central location for Drupal to update the asset database during the cronjob.
11. Active Directory Integration – The first step was to ensure the apache server itself was a domain member, which we accomplished through the standard samba/winbind configurations.  We then setup the PAM Authentication module which allowed the Drupal login to make use of the PHP PAM package, which ultimately allows it to use standard Linux PAM authentication – which once integrated into AD, includes all AD accounts/groups.  A little custom coding was also done to ensure matching Drupal roles were created for each AD group a user was a part of – allowing us to control access with Drupal (see #1 above) via AD groups.

There was a liberal dose of code within a custom module to glue some of the pieces together in a clean fashion, but overall the system works really smoothly, even with heavy use.  And the best part is, it consists of mainly free software, which is awesome considering how much we would have paid had we gone completely commercial for everything.

Please feel free to shoot me any specific questions about functionality if you have them – there were a number of details I didn’t want to bog the article down with, but I’d be happy to share my experiences.

Overview

One of the more daunting tasks of managing a larger network is keeping track of all your devices – both physically, and from a network monitoring perspective.  When I arrived on the job 3 years ago, the first major task I laid down for myself was implementing both an asset management system, as well as a network monitoring system, to ensure we always knew what we had, and if it was functioning properly.

I decided almost immediately that Drupal was the right candidate for the job of asset management.  There are a number of commercial IT/helpdesk systems out there which work great, but they are usually fairly expensive with recurring licensing costs, and my history with them has always been shaky.  Plus, I find myself not always using all the functionality I paid for.  I knew with my Drupal experience, I could get something comparable up in almost no time – this is not a discredit to IT packages, but moreso the power of the Drupal framework.

Network Monitoring – Cue Zenoss

Now that I had the hardware DB taken care of, I needed a NMS for monitoring.  Originally I was planning on Nagios, but a contractor who works for us (now friend) had introduced me to Zenoss, another open source alternative.  Zenoss is awesome – is absolutely has its quirks, and is not the most intuitive system to learn, but once things are up and running it’s great – and tremendously powerful.  So the choice was made.

Now – I had both pieces, but I absolutely hate entering data twice, and the interoperability guy in me loves integrating systems.  So I decided to write a script that would sync our Drupal database with Zenoss.  Drupal would serve as our master system, and any hardware we entered into it would automatically port over to Zenoss.  Any changes or deletions we made (IP address, location, name, etc) would sync over as well.

The below script performs this synchronization.  Some warnings up front – I’m not a Python guy by any means, I specifically learned it for this script, so I apologize for any slopping coding or obvious Python-y mistakes.  I’ve tried to thoroughly comment it to document how to use it and how it works.  Hopefully it can help some others out as well!

# Description: Sync devices to be monitored from Drupal to Zenoss
#
# Setup Work: Create a (or use an existing) content type that houses your hardware items to be monitored.
# They should have CCK fields for the IP address of the device, the name, and the type of
# device it is. The device type will determine the Zenoss class the script adds it to, and hence
# the kind of monitoring it will receive (e.g. Linux server, switch, ping only, etc)
#
# Additionally, in Zenoss, create a custom property field that will house the nid of the Drupal
# node. This serves as the foreign key and will be used to link the item in Drupal to its entry in Zenoss
#
# Usage: This script should be run from zendmd, and may be run once or periodically. We run it every 20 minutes from
# a cron job.
# It will create new entries in Zenoss for items not yet imported, delete ones that no longer exist in
# Drupal (it will only delete ones that were originally imported from Drupal), and will update ones that have
# been updated (type, IP, location, etc).
#
# Note: Excuse all the extra commits - we experienced some issues with data not being saved, and I threw some extra in
# there - they're almost definitely not necessaryimport MySQLdb

# Take a taxonomy term from Drupal identifying the type of monitoring to be done,
# and convert it to the appropriate Zenoss class path. Update these to whatever terms
# and Zenoss class paths that make sense for your environment. We setup ones for
# Linux and Windows servers, switches, waps, UPSes, PDUes, etc, as can be seen.
def getClassPath(passType):

if passType.lower() == "windows":
return "/Server/Windows"
elif passType.lower() == "linux":
return "/Server/Linux"
elif passType.lower() == "switch":
return "/Network/Switch"
elif passType.lower() == "mwap":
return "/Network/WAP/Managed"
elif passType.lower() == "uwap":
return "/Network/WAP/Unmanaged"
elif passType.lower() == "ups":
return "/Power/UPS"
elif passType.lower() == "pdu":
return "/Power/PDU"
elif passType.lower() == "camera":
return "/Camera"
elif passType.lower() == "cphone":
return "/Network/Telephone/Crash"
elif passType.lower() == "sphone":
return "/Network/Telephone/Standard"
elif passType.lower() == "printer":
return "/Printer"
elif passType.lower() == "converter":
return "/Network/Converter"
elif passType.lower() == "ping":
return "/Ping"
return "/Ping"

# Connect to Drupal's MySQL DB (Replace these values with the appropriate ones for your system)
imsConn = MySQLdb.connect(DRUPAL_MYSQL_SERVER, MYSQL_USER, MYSQL_PASSWORD, MYSQL_DB)
imsCursor = imsConn.cursor()

# Execute the query to grab all your items to be monitored. In our case, we have a node type called "hardware" that had CCK fields identifying the IP address,
# the type of hardware (a taxonomy term that dictated the Zenoss class of the item - see getClassPath above), a physical location, etc.
# You'll want to change the specific table/field names, but the inner join will probably stay, as you'll want to grab both the node and CCK fields that belong to it.
imsCursor.execute("""
SELECT node.nid, content_type_hardware.field_hardware_dns_value, content_type_hardware.field_hardware_location_value, content_type_hardware.field_hardware_ip_value, content_type_hardware.field_hardware_monitor_type_value, content_type_hardware.field_hardware_switchname_value, content_type_hardware.field_hardware_switchport_value
FROM node
INNER JOIN content_type_hardware ON node.nid = content_type_hardware.nid
""")

# Loop through all returned records - Check for additions, changes, and removals
while (1):
#tempRow is our current hardware item record
tempRow = imsCursor.fetchone()
if tempRow == None:
# No more entries, break out of loop
break
else:
# Search Zenoss records for the nid of the hardware item. A custom field will need to be created in Zenoss to serve
# as this foreign key. In our case, we used MHTIMSID - but you can use anything you'd like - just be sure to create the field in Zenoss.
found = False
for d in dmd.Devices.getSubDevices():
if d.cMHTIMSID != "":
if int(d.cMHTIMSID) == tempRow[0]:
found = True
break

if found == False:
# Hardware item not found, add it if it's monitored
if tempRow[4] != None:
dmd.DeviceLoader.loadDevice(("%s.yourdomain.com" % tempRow[1]).lower(), getClassPath(tempRow[4]),
"", "", # tag="", serialNumber="",
"", "", "", # zSnmpCommunity="", zSnmpPort=161, zSnmpVer=None,
"", 1000, "%s (%s - %s)" % (tempRow[2], tempRow[5], tempRow[6]), # rackSlot=0, productionState=1000, comments="",
"", "", # hwManufacturer="", hwProductName="" (neither or both),
"", "", # osManufacturer="", osProductName="" (neither or both).
"", "", "", #locationPath="",groupPaths=[],systemPaths=[],
"localhost", # performanceMonitor="localhost',
"none")
tempDevice = find(("%s.yourdomain.com" % tempRow[1]).lower())
tempDevice.setManageIp(tempRow[3])
commit()
# Save nid to Zenoss record (to serve as foreign key) for syncing
tempDevice._setProperty("cMHTIMSID","MHTIMS ID","string")
tempDevice.cMHTIMSID = tempRow[0];
commit()
else:
# Hardware item found - delete, update, or do nothing
if tempRow[4] == None:
# Delete if not set to monitor
dmd.Devices.removeDevices(d.id)
else:
# Update DNS and IP to current values
if d.getDeviceName() != ("%s.yourdomain.com" % tempRow[1]).lower():
d.renameDevice(("%s.yourdomain.com" % tempRow[1]).lower())
if d.getManageIp() != tempRow[3]:
d.setManageIp(tempRow[3])
commit()

# Change class if not set to "Manual" (We setup a taxonomy term called "Manual" that would turn off automatic Zenoss class selection during syncing
# and allow us to manually specificy the class of the device.
if tempRow[4] != "Manual":
d.changeDeviceClass(getClassPath(tempRow[4]))
commit()

# Update comments (location change)
d.comments = "%s (%s - %s)" % (tempRow[2], tempRow[5], tempRow[6])
commit()

# Save any missed changes
commit()

# Close connection to database
imsCursor.close()
imsConn.close()