Open Source software evaluation model

One of the best ways of being able to pick the right Open Source software by following the Open Source software evaluation model. This will greatly improve results that are based on what is going to be beneficial to the needs of you and your organization.

Community – the driving force behind an Open Source project

Release Activity – showing the progress made by the developers

Longevity – how long the product has been around

License – is one of the general Open Source licenses used

Support – from the community as well as paid support options

Documentation – user manuals and tutorials, developer documentation

Security – responding to vulnerabilities

Functionality – testing against functional requirements

Integration – standards, modularity and collaboration with other products

Goal and Origin – why was the project started and what is the current goal

These criteria form the key terms of the model. The evaluation process is described using these criteria. The practical part of the model consists of two steps. In the first step selection on the candidate list is performed, using four of the above criteria:

Functionality, Community, Release Activity and Longevity.

These criteria were selected because they can be evaluated quickly for each candidate in order to eliminate non-viable candidates and select the best ones. This step results in a ‘short list’ of candidates that can be evaluated in depth in the second step, taking a closer look at the software and the project using all ten criteria.


One of the most important aspects of open source is the community ”

The user community of an open source project consists of the people that use the software and participate in some way. One way of participating is by filing bug reports, which is a report on a problem in the software. This could be something very small, like a typing error in the text, or something large, like a feature that does not work properly. Another way is giving feedback on functionality the user would like to be added. Some users turn into developers over time, and participate in things such as fixing bugs or adding functionality themselves. They cross the line to the developer

community, which is often a line made very thin by encouraging participation and making the developer community accessible to anyone who is interested. In some cases the user and developer community interact fully in the same discussion areas.

The community of an Open Source project is very important because it is the community that does most of the testing and provides quality feedback. Instead of using financial resources to put the software through extensive testing and Quality Assurance (QA), like a proprietary vendor will do, the Open Source projects have the community as a resource. The more people are interested in a project, the more likely it is that it will be active and keep going. A large and active community says something about the acceptance of the software. If the software was not good enough to use, there would not be so many people who cared about its development

Release activity

The activity level of a project consists of the community activity and the development activity. The community was discussed above. The development activity is reflected in two parts:

• Their participation in the community

• The development itself – writing or changing the source code.

The latter activity is reflected mostly in the release activity. All software projects release new versions after a period of time. The number of releases per period and their significance, meaning how large the changes are per release (i.e. are there feature additions or just bug fixes in the release), illustrates the progress made by the developers. This gives a good indication of how seriously the developers are working on the software.


The longevity of a product is a measure of how long it has been around. It says something about a project’s stability and chance of survival. A project that is just starting it usually still full of bugs. The older a project, the less likely the developers will suddenly stop. But age is not always a guarantee of the chance of survival. First of all, very old software may be stuck on old technologies and methods, from which the only escape is to completely start over. Some software has already successfully gone through such a cycle, which is a good sign in terms of maturity. One thing that needs to be taken into account when products are not very young is whether or not there is still an active community around it.


Licenses in the Open Source world reflect something of the culture. The most important term in this context is ‘copyleft’, introduced by Richard Stallman, where copyright is used to ensure free software and their derivative works remain free. In essence a ‘copyleft’ license obligates anyone who redistributes software under that license in any way or form to also keep the code and any derivative code under the license, thus making any derivatives Open Source as well.

The most well-known example of a ‘copyleft’ license is the GNU GPL. This is also one of the most used licenses.

There are two types of support for a software product.

• Usage support – the answering of questions on the use of the software

• Failure support or maintenance – the solving of problems in the software

Often the two get mixed at some level because users do not always know the right way to use the product. Their support request will start as a problem report and later becomes part of usage support.

Aside from community support, larger or more popular projects may have paid support options. The software is free to use, but the user has the option to get professional support for a fee, either on a service agreement basis where a subscription fee is payed for a certain period of time, or a per incident fee for each time the user calls on support.

The fact that paid support exists for a Open Source product, especially third party support, is a sign of maturity and a sign the product is taken seriously.


There are two main types of documentation:

• User documentation

• Developer documentation

User documentation contains all documents that describe how to use the system. For certain applications there can be different levels in the user documentation, corresponding with different user levels and rights.

The other main type of documentation, which plays a much larger role in Open Source software than in proprietary applications, is developer documentation. A voluntary decentralised distribution of labour could not work without it. The developer documentation concerns separate documents on how to add or change the code, as well as documentation within the source code.

A third type of documentation that is often available for larger server-based applications is maintainers documentation, which includes the install and upgrade instructions. These need to be clear, with the required infrastructure and the steps for installing the software properly explained.


There are various security advisories to check for bugs in all types of software that make it vulnerable to attacks. A couple of well known advisories are securityfocus and secunia. Check these sites for vulnerabilities in the software and see how soon they were fixed in a new version. Keep in mind that more popular software will have a higher chance of having vulnerability reports, so the mere lack of reports is no proof of its security. On the project’s website it can be seen, for instance in the community or in release notes, how serious the project is about security.


One problem with Open Source projects is that the documentation is not always up to date with the latest software. Look beyond the feature list on the website to find out what features the software has.

Eventually the software itself should be investigated. If it is a web-based application, an online demo might be available, though installing it on a test environment could be useful because it also gives insight on how well the software installs.

A list of functional requirements for the goal of use of the software can be used to check if the needed functionality is available. It is wise to make a distinction in the list between features that are absolutely necessary, where the absence would lead to elimination, and those that would be a plus, which results in a higher score. If there is something missing there is always the option to built it or have it built.

When comparing functionality, those features that are part of the functional requirements should take priority. While evaluating the software, features may be found in some of the candidates that are very useful for the goal.



  • Check the documentation and code base for evidence of a modular structure. The development documentation describes how to add modules if the software is modular.


  • Software projects that use standards usually include information about this in their feature list or documentation. Check whether the standards implementation is complete or still underway.

Collaboration with other products

  • If the software can work with relevant other product this can usually be found in the feature list or documentation.

Software Requirements

  • Check the software requirements in the documentation of the software. There is usually a section on this in the installation document. Check whether these requirements can be met in the current environment , and how flexible they are in terms of the versions of the required software.

Goal and Origin

Something about the history of the project can often be found in the documentation or on a special page on the website. See if the goal of the software and the reason it got started is compatible with the intended use.

This entry was posted in Linux, Macintosh, Windows and tagged , , , . Bookmark the permalink.

2 Responses to Open Source software evaluation model

  1. Polprav says:

    Hello from Russia!
    Can I quote a post in your blog with the link to you?

  2. macpablodesigns says:


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s