Deconstructing Software

Structure101 V3 Released, Adds Architecture Control for Teams

Released today, the new version 3 capabilities make Structure101 a nicely rounded architectural control solution in addition to the previous structural analysis and complexity measurement capabilities.

For example:

• You can now define layering constraints on your code-base using simple, intuitive architecture block diagrams
• Communicate these architecture diagrams to the development team through IDE plug-ins
• Developers get warned immediately if they make code changes that are inconsistent with the architecture
• RSS activity feeds let you know if new architecture violations make it into a build

Also, there's a new online demo (about 13 minutes, with audio (me!)) and the version 3 Help is available online.

Full press release

Enjoy!

Package design matters - Part 1

Java packages are often used like file-system folders to organize source. But source files differ from "normal" files in that they are highly inter-dependent. Considering this interdependence as a package hierarchy evolves can have significant productivity benefits.

Packages as Folders

Java packages provide an ideal way to organize code into a scalable, hierarchical structure that helps us find specific code.

In this sense, packages can be used like folders in a file-system: 

• We place files with something in common in the same folder. 
• When a folder grows too big and we find we’re having trouble finding files, we split the folder into sub-folders according to some criteria that makes sense to us.
• We share files by placing them in a common area on company network, in which case the structure evolves according to the varying criteria of different people.
• We often have trouble deciding which folder a file best belongs, and make an arbitrary decision.

Often Packages are only used as a kind of filesystem equivallent.  However the package hierarchy can also be used to reinforce the intended design and associated development activities.

Packages as Design Abstractions

Source files differ from other files typically stored in the file-system:

• They depend on the detailed contents of other source files. 
• Are created and edited in groups of multiple files. 
• Are subject to a high number of relatively small changes.
• Are edited by a team of developers rather than individuals. 
• Should be reusable on future projects.
• Should be easy to change without impacting other files too widely.
• Must support the defined deployment environment.
• Are subject to a QA, version control and release processes.

The aim of a package design should be to support these characteristics.  For example, the design could explicitly support Martin's “Reuse/Release Equivalence Principle (REP)” (article, book) whereby packages are developed, built, tested and released against released versions of the packages upon which they depend. 

Design is not something that happens once at the start of the project – it is an activity that spans the life of an application or product.  This fact has become explicit with the iterative and Agile development models.  As the code-level design continues, the package-level design emerges. 
Unfortunately, the emergent design is often invisible and so forgotten.  Not only does the original design degrade, but the overall structure tends to become excessively complex.  As the supporting structure dissolves, development activities become harder and the cost of each new feature increases.

This priciple of emergent design is important here. Clearly when I have a project of 50 classes in a half-dozen packages, the overhead of a sub-optimal package dependencies isn't going to slaughter me. But if my project is going to grow to 5000 classes, then putting in minimal effort from the start can save huge effort when things get more complicated.

In part 2 I'll take a closer look at cyclic package dependencies and why they matter.

Code Organization Guidelines for Large Code Bases

In an excellent on-line presentation Juergen Hoeller gives rationale and guidelines for controlling the structure of large, evolving code-bases. Juergen is the chief architect of the Spring framework, which as I have previously pointed out is structurally almost perfect. This didn't happen by accident.

If you don't have time go though the 88 minute presentation, here is a nice sysnopsis by Mike Nereson.

Structure101 Supports Java 1.5

We have overhauled the byte-code parser to now pick up Java 1.5 constructs (e.g. generics and annotations), plus it is much faster.

You can download the first build that includes the new parser here.

If you currently use the structure101 IDE plug-in you may find that you get bogus warnings on "new" architectural violation when you start using the new parser. This is  because the parser is now finding a few more dependencies related to the Java 1.5 language updates, and thinks they are new compared to your reference snapshot. If it's giving you a headache, you should just republish your reference snapshot to the structure101 repository with the new parser.

Eclipse Plugin (OSGi) Visualization

If you are going mad trying to figure out the dependencies between lots of Eclipse plug-ins, or work with other large OSGi systems, you may be interested in this.

We've had a few people looking for an Eclipse/OSGi backend for Structure101, and with all the hype about OSGi lately, we decided to lift the lid on it.  Here is an early version that you can download. If you're an Eclipse user, just point it at your plugins directory to see the same kind of views, hierarchies and slices that you get with the Java version.

Below is a random screen shot of my Eclipse plug-ins to give you the idea (click for the full-size image).

It's pretty rough at the moment - the download page indicates where we've made some arbitrary decisions on the model structure and where we think it's probably not quite right. I'd love some comments from OSGi heads on if/how you'd like us to change it to make it more finished. If you think you'd find it useful, talk to Paul about an extended license key.

Structure101 v2 goes GA today

Additions let you see complete slices of a code-base at any level, home in on structural complexity, view dependency graphs in matrix form, and map code items and groups (like tangles) through different hierarchies, slices and perspectives (more download).

Structure101 v2 beta 2 released

Structure101 is for contolling architecture and structural complexity. Version 2 introduced slicing and tagging. Beta 2 adds dependency hiding and cross-perspective navigation.

New in v2 beta 2:

• Context menus (right mouse click) have been added. This introduces the new capability to navigate easily between views (perspectives).
• It is now possible to tag dependencies as well as items.
• It is also possible to hide tagged dependencies – I.e. remove them from the model. This is useful for simulating structural changes.
• Performance problems on Linux and Mac have been fixed.
• Look and feel on Linux and Mac have been improved

Spring 2's architecture - A single dependency cycle slipped in

The Spring guys have let a single dependency cycle into their architecture. A very small flaw, but it's a perfect example of why you need to check your code-base at different levels to keep it truly tangle-free.

I did a quick analysis of the Spring Framework some time back and sure enough found their claims of a cycle-free architecture to be correct - a pleasure to behold!

The recent announcement of Spring 2.0 rc4 prompted me to point Structure101 version 2 at same and check they were keeping up the high standard they had set themselves. The Structure101 "notables" quickly took me to the org.springframework.aop package which contains the following tangle:

Ok, this is not exactly a fatal flaw, far from it, but it surprised me because I know that Juergen keeps an eye on this stuff. Then I took a look at the leaf package slice (leaf packages being packages that contain classes), and guess what? Not a single tangle.  It is only when you look at the slice one level up that the tangle is apparent:

Taking a look at the leaf packages contained by these 2 packages:

(The package names are relative to org.springframework.aop). The dependency between the tagged packages (blue dots) is the one causing the problem. Overlaying the parent package boundaries on this graph, you can see why it is that, although the package diagram is acyclic, dependencies between the parent packages go in both directions, making them cyclically dependent.

I presume they check only at the flat-package level, which is why this one slipped through the net.

Tracking complexity debt

Un-monitored, the complexity of a code-base increases with its size. Jboss and Struts are perfect examples. However monitoring complexity helps you keep complexity debt under control, or even down to zero.

If you publish the last couple of years worth of releases of your project to a Structure101 repository, you'll probably see something like this in the Structure101 Tracker web application:

jboss over time

Structure101 matches the amount of XS to the lines of code that cause it. Unless someone pays attention to it, the same team will tend to code-in a consistent degree of complexity debt as they go - in this case they're running at a not-unusual but alarming 80%.

Struts shows a similar chart running at about 57% XS:

struts over time

I was in with the Prime Carrier guys yesterday. They've been tracking their excess complexity for several months now and it really shows.  Here's the chart for one of their projects:

Prime Carrier project 1 over time

As you can see, XS is hugging the zero line even as the size of the code-base grows. This is why we call it "excess" (XS) - it really is excessive and totally avoidable. Occasionally a cyclic dependency (tangle) or fat class may get into the build, but it's flagged and usually fixed during the next iteration.

They recently released a new version of another product under severe deadline pressure and consciously decided to pick up a bit of short-term debt. As you can see, they're already paying it back before it incurs too much "interest":

Prime Carrier project 2 over time

These are a great bunch of hard-nosed developers who are keeping the code-base clear of debt so they can focus on the principal - customer requirements.

Manage complexity like debt

Ben Hosking writes in Managing Complexity - The aim of Designing Code that:

"The most important part of design is managing complexity"

I like the simplicity of that. What happens if you don't manage complexity. Well, it starts to cost. Talking at OOPSLA 2004, Ward Cunningham (Mr. Wiki) compared complexity with debt:

"Manage complexity like debt," Cunningham told attendees. Using this analogy, he likened skipping designs to borrowing money; dealing with maintenance headaches like incurring interest payments; refactoring, which is improving the design of existing code, like repaying debt; and creating engineering policies like devising financial policies.

In an interview with Bill Venners (Artima), Andy Hunt (Pragmatic Programmer) extends the analogy concisely:

"But just like real debt, it doesn't take much to get to the point where you can never pay it back, where you have so many problems you can never go back and address them."

It's a lovely metaphor. But it does breaks down in one place. Project managers don't get a pile of bills through the door every month. Even if they wanted to, they can't rip them open, sum them up, compare them against income and outgoings and discover just how fragged they are, or even hell, that they can afford loads more debt!

Well it's not quite that bad. We can at least measure and sum up the complexity of items at different levels of design breakout (methods, classes, packages, subsystems and projects).  We may not be able to put a hard complexity number on the tipping point (insolvency), but we can give you a number. With this you can compare projects, monitor trends that show where it's getting more or less complex, and discover which items at what level are causing the trend.

For example here is the home page for the Structure101 Tracker web application showing the sizes and over-complexity of several projects:

Now, correlate XS with the depth of furrow on team leaders' foreheads, and you've really got something to go on...