In Progress

It is amazing to see all of the great stuff going on here at DevJam.  After getting the RESTful stuff into the code on Tuesday night, Craig Miskell and Alejandro Galue took off like a shot adding RESTful interfaces to all or our main objects.  Things were going so quick that we had to stop and take some time to review together the URLs we wanted use so we were consistent with things.  We sat together along with Mike Huot, Rob Moore and a few others that stayed in or out and came up with a set of URL that will describe how to access OpenNMS model objects via REST.  We posted the results on our up on the wiki so everyone could see them and start contributing.  After we get these done we just need to go through and hit the configuration work.  A totally awesome day even though I personally hardly did anything.  These guys rock and just like Stymie from the Little Rascals… I can’t keep up!

The second day of DevJam turned out to be a great day. I fixed the spring problem I was having very early. It turned out to be a old spring lib sneaking into the installation. Those sneaky libraries.

Next up was the jersey integation, but not before a few detours to work with a some of the other guys. I spent some time getting Rob Moore building and such. What a cool guy he is! I sure hope he likes hanging around enough to help us with some of the web ui enhancements we’ve got planned. Currently he’s working on putting some AJAX swizzle on some of our tables and lists. I won’t spill the beans about that ’til we get it working.

After that it was on to visit with DJ Gregor and Mike Huot. After I introduced the RESTful ideas for the webapp Mike Huot thought that a Graph Definition editor might be a handy addition to OpenNMS. YES!!!! So they’ve been working out how to get that in. Just need that Jersey stuff integrated so they can start using it.

So on to the Jersey integration, oh, but first I’ll talk with Ben Reed about the remote poller – webstart poller de-duplication. Nice work on that. Looks like there will just be one now and it’ll be deployable in any of the old ways. The cool part is, that it will keep itself up to date with the server.  Ben rocks.

Ok, so now for that jersey integration. Oh wait, got to to stop and find out what Alejandro Galue has been working on. Hey Alejandro, that looks like a RESTful resource. Holy cow! Alejandro did the Jersey integration for me! Sweet! I knew that expenses app would come in handy. Worked with him and got all of his stuff merged to trunk just in time for bed!

That sure was a great day. And I didn’t even mention the other cool stuff that happened. Like the great talk Alex Finger and I had about the World Wars on the way to lunch and the fun we had at the churrascaria. Oh and don’t forget to with DJ a belated Happy Birthday if you see him today.  Oh yeah, and I know I saw an AIR client to opennms at one point or another.

Man, this stuff is getting better every year and to think, its only Day 2!

Well, the first day of DevJam is over and as usual I wish I had more done. Been fighting getting opennms trunk converted to spring 2.5. Seems to be complaining about creating some beans in the availability code. Heh. Not sure why though, haven’t even changed and of that. Oh well, still a bit before bed, maybe I’ll it get working. Night!

Have I ever mentioned that I have way TOO much to do? Anyway, I did quite a bit of good work this week in preparation for DevJam in Atlanta but as I get my gear together and wait for the guys to pick me up in the next hour or so I realize just how much I have left.

I started last week with the hopes of putting together a series of blog entries on using REST as the OpenNMS Web technology of the future. Unfortunately, as it always is with programming, I just wasn’t able to get it all finished. I did discover a TOTALLY cool API for calling RESTful services for Java. This API is actually part of the Jersey project that I am using for the server side. I know its foolish to think I’ll get a lot of work done on the drive down to Atlanta, but maybe I will. The main things I really need to do to help move DevJam along are

1. Move OpenNMS Trunk to springframework 2.5. This will make some of the RESTful work and other development work much easier
2. Decide how I’m going to integrate the Jersey servlet into the OpenNMS. Should I make an entirely new webapp or should I make just make it /opennms/rest? That will definitely need to be decided before I get there? (I think I’ve decided just be writing this. Any bets on which it will be?)
3. Create a list of the ‘Resources’ that will be represented in the web UI and assign URLs to them. This one is probably the most important because it goes modeling data to the team and you all know how I feel about modeling. I can probably get away with just a reasonable subset of this but there really needs to be a sane description of what kinds of things the OpenNMS app shows and what the standard web address for them is.

That seem like too much to do on the drive to Atlanta? Probably. Have I ever mentioned that I have way TOO much to do?

Anyway! See you at DevJam!

Over the last several years I’ve been doing a lot of work on OpenNMS and I’ve changed a lot of different things. I’ve changed the build system, the poller, the collector, and the way the database is accessed. Through all of this there has been one perpetual thorn in my side and that has been the OpenNMS web user interface.

So what are you so afraid of?

The real issue with me and the webapp has been that it just feels very difficult to change. Whenever I have any non-trivial work to do on the webapp I cower with fear of getting into some pitt of javascript, servlet, or jsp scriptlet hell, not to mention the gobs of java code that I have yet to understand.

But the real reason that I’m afraid of the OpenNMS webapp has been the problem of OpenNMS since I started with it. The original OpenNMS code base had no domain model. Rather than simply asking the peristence layer for some domain objects, making a few changes and then saving them again, jsp and servlet code had been forced to dive into JDBC directly embedding queries left and right, storm thru the file system looking for configuration and data files, and representing this data as ints, and strings or some cobbled together object that only makes sense for that single part of the application.

On top of this, the lack of the model has made the webapp as a whole seem somewhat cobbled together. The viewing pages aren’t horrible but the administrative stuff is very confusing. And the graphing stuff just never seems to be quite right.

Another by product of missing a model is that OpenNMS just has way too much code. How many times do I need to write the code to process a configuration package. How many times do I need to build the same queries and result set processing to get the information I want. How many different times will I pass the same three parameters, nodeId, ipAddr and serviceName and not turn it into an object that represents the actual MonitoredService.

All of this makes the code very fragile. A change in one place leads to bugs in others because everyone seems to depends on the details of everyone else’s work. At the same time my relectance to really make any changes to is like an anchor pulling OpenNMS down. What good is cool data collection when creating interesting graphs is so hard? How can I consider collecting topology related data when there’s no clear way to show it to the user? On Call Schedules, User Pages, ACLs, Business Process Monitoring, Executive Reporting, Improved Maps, Users who can change their own passwords. All of this and more has been hindered by my perpetual dread of mucking around in the web app.

Programming is not for cowards

After working for years on OpenNMS if there’s one thing I know for certain its this. Programming is not for cowards. So why haven’t I undertaken the charge to overcome this menace of Java debauchery? Well the answer is simple. I haven’t had a decent plan. Sure I’ve made a few feeble attempts to test the waters in this area but known would be considered any sort of success. Spring MVC was the biggest effort along with some others like GWT, Wicket, and I’m sure we all remember the Secret Project.

Many of those failed not because they are inherently bad technologies but that they only assisted you in coding rather than in thinking. The failing of the webapp has not been a lack of technology but a lack of a clear design in terms of modeled objects. Instead of a technology that helps as parse form data, convert strings to dates, or write fancy javascript (though we really want these) we need something that will guide us in modeling the concepts behind our interface.

We want the user of the web ui AND the developer to be thinking about the problem in the same way. When this occurs then the important concepts that the users sees will be matched by objects and interfaces in the code making the transition from user to contributer simpler and making the learning curve of OpenNMS development simpler. Not to mention making the code easier for all of us.

Real Ultimate Power

So… all that sounds nice and all, but where will we find a technology or strategy that will do so much? For some time I had wondered the same thing. Until now.

Due to a serendipitous string of events that included Jason Aras, free books, Red Bull, Tarus Balog, Texas, new feature requests that I’m still behind on, David Hustace, car pooling, and yet another importer tweak, I was introduced to a web architecture known as Resource Oriented architecture which is what many people have come to mean when they say a RESTful architecture. (Please see “RESTful Web Services” for more.)

The key principals of this architecture are:

1. A webapp is really made up of a collection of ‘Resources’
2. Each of these resources has an address in the form of a URL.
3. There is a very simple set of operations available on these resources known as the ‘Universal Interface’- GET, POST, PUT, DELETE.
4. Information sent back and forth in a URL is a representation of the state of one of these resources.

You are probably tempted to do like I did and brush off this seemingly simple list as “Yet Another Set of Coding Standards” or something like that. If you do then you’re making a big mistake.

Simple as this list is, if we embrace its wisdom, it will lead to a remarkably simpler OpenNMS. The above principals will lead us to clean modeling of our domain, push us to proper layering of the application, enable simple intergration with other technologies, and best of all, make web ui development a joy.

I will be posting a series of blog entries over the next several days that I hope will convince you that this is going to simplify OpenNMS web development for good. I hope to introduce the technologies and processes that we’ll be using, implement a simple example, and then document the first steps of OpenNMS webapp design.

Please stay tuned

I’ve started refactoring the actual SnmpCollector code in an attempt to simplify a few things as a prepare to reorganize. I figured I’d start at the beginning. WHOA! Have you seen the initialize method in there. Check it out in 1.2! This one methods was hundreds of lines longs generated and/or manipulated at least 3 Maps, piles of Lists, and on top of that dove into the database for a number of different things. In one case it even made database calls inside a loop processing a result set! Needless to say when I look at it I was completely overwhelmed!

Fortunately for me I had just been discussing refactoring with Dave and Mike Huot. I had been going through the examples from the refactoring book and had been showing Dave some of the work I had done on Collectd. I had noticed a comment that said [paraphrased], “If you run across a very complex method with large number of local variables, turn it into a method object and then refactor.” I had been a little skeptical with respect to how this would really help but when I came across this method I decided I would give it a try.

I created an object called Initializer. The purpose of this object was just to have one public method called execute that did all the same things as the ‘initialize method did. I created an instance of the object and moved the method into the Initializer class using Eclipse. Next I went through the object and, again using Eclipse, I made all the locals into fields. This at first doesn’t really do anything to help the complexity of the code but it makes the next steps possible which make things MUCH easier.

Once all of the locals are fields, Then it is a simple matter to go through the code and extract methods that perform various useful functions. By simple doing this I was able to turn the initialize into the following:

	initialize();
	/*
	 * All database calls wrapped in try/finally block so we make certain
	 * that the connection will be closed when we are finished.
	 */
	allocateDBConn();
	try {
		getPrimarySnmpInfo();

		getSysObjId();

		createNodeInfo();

		determineSnmpVersion();

		getSnmpInfoForInterfaces();

		verifyCollectionIsNecessary();

	} finally {
		closeDBConn();
	}

	logCompletion();

The code in these methods is still quite ugly but now I have a number of methods with a much more easily defined purpose. I am also able to do the same thing to those method as I need to. The ultimate goal of course is to turn most of this function into operations on the appropriate objects as this makes things MUCH simpler. I’m not finished with this object yet but I suspect by the time I finish this object will ‘appear’ to do nothing more than validate the parameters to the initialize method.

This refactoring thing just keeps getting better and better. But knowing refactoring and actually doing refactoring sure are very different and real mastery comes with practice. I sure wish I had this rectoring skilll when I worked on the poller!

Well, happy refactoring!
Matt

Well, The last day or two has been quite a challenge. I’ve made some significant progress on the refactoring of collectd, but got held up for two key reasons. One is that I had gotten past the completely mechanical refactorings to the more sophisticated and I realized that the tests I had (or didn’t have) were not going to do it. Soooo… I had to start figuring out how to write some collectd tests. The second hindrance to progress was the fact that I had been ‘pussy footing’ around the ConfigFactories. They all but stopped my efforts to write tests and were making the refactoring difficult because some of the functions in the factory need to be on other objects.

So all this to say that today, I started refactoring the CollectionConfigFactory to simplify it. This effort has been very educational and has helped to solidify in my mind exactly what a ‘configuration DAO’ needs to look like and how it should be implemented. Hopefully I will be able to complete this soon as this is a prerequisite to abstracting the RRD code out.

Anyway, stay tuned as there is much more to come.

Matt

Well I didn’t make it on Thursday and I had Friday off for the holiday weekend. So it’s now Tuesday and I’m finally getting back to things. I spent some more time Monday working on Collectd and feel like things have gotten quite a bit simpler in there. I’ve been working primarily with Collectd.java and CollectableService.java trying to ‘find’ a good set of objects that describes what is going on in this set of code. I’ve introduced a few objects that make the code seem quite a bit simpler. Still need to work on the event handling though as well as the scheduling. THEN there’s SNMPCollector. That will be where the fun really begins.

Oh yeah, I also wanted to mention that I found a few places in the main collector that assume we are only collectiong SNMP data so I’ll have to figure out what to do about that.

Anyway, more to come,

Matt

Well, mhuot and i had a great time pairing this morning. It was one of the more product pairing sessions we’ve had in a while. A while back Dave and I implemented the EventTrasnlator service that can create ’standard’ kinds (or any kind actually) of events from other events such as snmp traps. This is very handy for setting up notifications about unusual things or things that are difficult to monitor directly with OpenNMS. One of the features we added was the ability to put SQL queries in. Mike wanted to be able to define databases other than the OpenNMS database and use queries against them to aid in the translation of events. This is a great idea! In order to help do this Mike and I added pretty simple multi-database support to the DatabaseConnectionFactory. This will make it simple for a user to configure a new database into opennms and for the EventTranslator to use it to make queries.

After that I merged a few fixes into a branch a for Chuck Thier at Rackspace and then up to HEAD from 1.2. These were JDBCMonitor and JDBCPlugin stuff from Tobey Pasheilich along with a Pllugin/Monitor pair for calling a stored procedure in the database. Pretty cool actually.

Heading off to the North Carolina Zoo tomorrow.. So see y’all on Thursday!

Good Job today Mike!
Matt