The Development Process for NetCDF

What Management Style is used for the NetCDF Project?


NetCDF is a very successful free software package. It's used by NASA, NOAA, the ESA, and the IPCC, to list just a few. It's a global standard for weather and climate data, and also used in other sciences.

As such, the netCDF library is a critical piece of infrastructure for many very important software systems. It has performed reliably and well for more than 20 years, yet it has not remained static.

In the last 20 years new features include NetCDF-Java and a whole ecosystem of Java data tools, NetCDF-4 and HDF5 integration, remote data access with OPeNDAP, and the addition of large-file capable binary formats 64-bit offset and CDF5.

NetCDF continues to grow and evolve with the hardware and software that make up science data processing systems. It is as relevant for cutting edge computer science today as it was when introduced

You may well wonder about the development process used for this very successful project. You may also be amazed to learn that the netCDF team is very small - generally less than 2 full time people.

Demi-Gods of Programming

The development processes and environment under which netCDF is developed (along with many other successful tools), is the work of one mad programming genius, Russ.

Russ' madness is well-documented, but few are aware of his contributions in the field of software development management.

Russ understood that some programmers are much more productive than other programmers. He conceived of the Unidata program center as a mountain-full of programmer demi-gods.


The Rules of Demi-God Programmer Management


I will distill Russ' ideas on programmer management into a few rules:

* everyone gets an office
* everything other than programming is optional
* if someone tells you to do something stupid, it's never your job to do it
* everyone else is too busy to tell you what to do or how to do it
* automation is always good

An Office for All

If you are programming, you need to concentrate. If you are being interrupted or bothered, you are not concentrating, therefore not programming.

Therefor every programmer needs an office. No exceptions.

This is a crucial productivity booster. When programmers are doing heads-down coding, in a quiet environment, and are not interrupted, a lot more gets done.

You're a Programmer, Nothing Else 

The dress code is "the holes in your pants can't match the holes in your underpants."

A similar attitude was taken to most other aspects of the workplace. Whatever is not helpful for programming, is optional.

Monthly meetings were held and attendance was mandatory. Otherwise, every meeting was optional.

Don't Do Stupid Stuff

If you think something is stupid, don't do it. Do something else instead. Never do something stupid just because you were told to do so. Even if the person who told you is your boss, they don't really mean for you to do something stupid. So don't.

Figure Out What to Work On

There are so many challenges with science data! There are more scientific instruments now than at any time in history, and they produce more data than has ever been produced before.

If you can't figure out how to help, don't expect someone else to be able to tell you. Take a look at the problem space, and start coming up with solutions.

Automation is Our Friend


Anything we can automate is something we should automate. Unidata was using automated testing long before most organizations, long before the concepts of agile programming were articulated. Automated testing provides good testing with much less work.

On a project with major goals and challenges, we need to automate as much as we can, so that we can focus our efforts on moving the project forward.

The Result

The result of hiring a few good programmers, and providing this management style, was netCDF, (and many other advances in software for Earth science research software, developed at Unidata).

I know this was not just netCDF, because the same style of management was applied to all Unidata projects and products, and they were often very successful.

Some projects didn't work out well, and sometimes the demi-gods could be difficult to deal with (just like the real demi-gods). But this style of management produced more quality software, with far less resourses, than I have ever experienced.





Comments

Post a Comment

Popular posts from this blog

Building NetCDF for HPC

Image
Building netCDF from scratch, on a High Performance Computing (HPC) platform is a challenge. There are a lot of other libraries involved. This diagram shows all the possible 3-rd party libraries in a netCDF build (the yellow boxes): Why Build from Source? Building the netCDF library from source is not usually necessary for most users. On most HPC systems, netCDF will already be install somewhere. Contact your sysadmins and ask how to use it. Sometimes, you need to build from source, because: You need the latest version, and the sysadmins haven't installed it. You need some combination of tools and versions that is not already installed. You are the sysadmin, and you have to build the libraries so that all your users don't have to. You want to have full control and understanding of the build process. Building with Autotools The libraries we need to build all have standard "autotools" build systems. This means that the developers use autoconf
Read more

The Different Binary Formats of NetCDF

Image
Under the covers, the netCDF library uses several different binary formats for data. It's important for advanced users to understand the different formats, and use the correct one. The Classic Format Originally there was just one format for netCDF files. It is described in detail in the netCDF documentation (including here ), but it's not necessary to understand the low-level details. Once we introduced the 64-bit offset format, we needed a name for this format, so we call it the "classic" format. The classic format can be read and written by any install of the netCDF library, even if all the bells and whistles are turned off at install. The classic format has limits which make it hard to use on data files larger than 2 GB. The 64-bit Offset Format In order to address the 2 GB limits of the classic format, the 64-bit offset format was contributed by a user, and introduced in netCDF version 3.6.0. The 64-bit offset format changes the classic format very
Read more

What is NetCDF, and When Should You Use It

NetCDF is a data format for science and engineering data. It's also a set of free and open software libraries and tools which support the format. NetCDF works better than a database for most science and engineering data, which usually does not fit well into the relational database model. NetCDF has been accepted for decades by the Earth science community to store and share weather and climate data. It is also used by other science and engineering communities. Unique Needs for Science Data Science data needs are different from the data needs of commercial entities, like Google or Amazon. Why Not Use Databases? Databases are all about tables of data. Each table is a two dimensional array of fields. For example, in a table of customer data at Amazon, they will have your customer ID number, which allows them to look up your record, and then they will have fields like "first name", "last name", "street address 1", "street address 2", e
Read more