Our monitoring team replaced their old servers with new servers last year and a new version. Unfortunately the servers aren't powerful enough to crunch through more than a few thousand alarms before taking a dump so they only save alarms for 48 hours. This makes it impossible for us to do any trending and management of alarms, especially since the monitoring software isn't capable of generating a report of just what it is monitoring.
On my own, I added a central email account to our group email account. Our group email receives the alarms from monitoring of course. This puts every alarm into a central mailbox. I started this in 2009 so I have about 350,000 alarms in various mailboxes (I roll them once a year).
Yesterday I parsed out relevant information from each email and imported all 350,000 alarms into my inventory database which associated the alarms with servers and gives me a long term view at the environment. I've set up an automated process that now parses each email and imports it into the database so we have accurate information and have our alarms for long term trending. I also whipped up a couple of scripts to provide the view into the alarms.
Currently you can only select a date range but today I'll be adding bits to display alarms on the individual server detail page and adding a search feature so you can filter information even further. I also will create a report with the number alarms for each server by year, etc, more of a management view of the alarming than the trending stuff we sysadmins want to do.
Part two is after 7 years of working on my Inventory program including the recent upgrade to 3.0, and after the "official" inventory conversion team (importing from the "official" inventory tool, which only manages physical assets) has refused over and over to use information that's in my "unofficial" "unapproved" inventory too, one of the VPs has instructed the team that my tool is much more complete and is the "database of record" for inventory and that they
will use my inventory data to provide all the details that the "official" inventory asset tracking database has.
So I'm writing scripts to extract data from my inventory tool in a format that can be used to import it into the new "Asset Tracking ITIL CMDB" system.
*wheee*
