This interface is the main access point to the release database and presents information about the components. Existing Tools release management. The tools support the development of accelerator control software and are being used by both specialized Java developers in addition to physicists developing operational software for the CERN Control Center. Chapter 5. The aim of this chapter is to provide a base of understanding before the presentation of the practical part of the thesis.
All new software would be written in Java, and when time and resources would allow it, old applications would be rewritten in the new language. To fulfill this need, it was decided to set up an easy and uniform process that should be used by all future developments. This process should be based on standards, knowledge and tools available in the Java community whenever possible.
The tools to support this process were decided to be implemented in-house, thereby allowing custom functionality, and avoid the risk of implementing a tool which would be discontinued in the future. CmmnBuild pronounced: common build was initiated to support tasks such as the building process and dependency management. The Release Tool 1 handles all the interaction with the software repositories. Before elaborating on the tools, the development process they support is explained.
The process of developing accelerator control software is defined as follows: All activities from the moment a developer starts a new project to the moment the resulting application is running on operational consoles in the control room The process includes issues such as project and source-code organization, build services, dependencies management, release management and application deployment.
The development process in this context does imply a certain development method. Rather, it refers to important tasks involved in the process of developing software. Following is a presentation of the key issues in the accelerator control software development process.
Projects and code organization As mentioned in the introduction, the development strategy of accelerator control software implies focus on small interdependent products, implemented, built, tested and released independent of each other. For such a strategy to work, the use of solid conventions and guidelines for source organization and structure is crucial.
A project is to be given a name corresponding to where it belongs. A dedicated repository for accelerator controls software is maintained with a well defined directory structure. See figure 5. More advanced functionality is integrated into CmmnBuild. One example is tagging. The development process of accelerator control software 27 Figure 5.
The build services are used for prototyping, testing of new functionality and creating a distribution before releasing a new version of a product. The build process includes actions such as compilation, packaging, running of unit tests and generation of certain files and documents. The build process is supported by the CmmnBuild tool. In order to simplify the dependency declaration for the user, and facilitate the maintainability of all products, it is crucial to support transitive dependencies.
This means that a user only specifies the products he depends directly on, without worrying about the libraries his dependencies rely on. In a scenario where a low level product changes its dependency-declaration, all products depending on this product directly or indirectly , would have to update their dependencies as well. This quickly becomes complicated, especially in environments with many products and complicated dependency-trees. CmmnBuild handles the management of dependencies and fetches both direct and indirect dependencies from either of the two product stores.
Like all three tools mentioned in chapter 4, CmmnBuild handles the different repositories transparent to the user. Release management Release management is the process of installing a built and tested product into the operational distribution area, referred to as the production repository.
All deliverables are installed and distributed on their corresponding platforms, where they are made available for use in operation.
The products are available through an assigned version number. Previous operational versions stay available. The release management is handled by the Release Tool. To enable the use of JaWS, a web server has been installed which makes the production and the third-party repositories accessible through the web.
It is possible to run any released version of the product. Also available is the generation of a command script for launching an application.
Presented here were the most important aspects of the development process, with an indication of which services is provided by which tool. Following is a detailed explanation of the two tools. First is a closer look at the operational environment in which the tools operate. Second is an elaboration of why and how the tools have been developed. The last section explains the technical aspects of the tools, how they interact and what are the services they provide.
The build and release tools 29 5. The developers of such software vary in background and software development experience. User Characteristics The users of CmmnBuild and Release Tools consists of both specialized Java developers and physicists developing operational software.
CmmnBuild and Release Tool have an estimated total of 80 users. The main example is Fermilab in the USA 2. Both of them can be used without the use of the other and the development of the two has previously been done separately. They are now combined to one project. The next two sections explains the choice of development for the tools.
CmmnBuild Before starting the development of an in-house tool to solve the need for configuration management of accelerator control software, several tools in the Java community were examined and considered.
However, they were all found to be in an early stage of development, with limited functionality, especially regarding dependency management. They were not considered mature enough, and there was no guaranty that these tools would be continued into the future. In addition, no satisfactory solution to the incorporation of 3rd party libraries was found.
Considering these facts, it was decided to develop an in-house, custom solution. This would guaranty the section the necessary functionality. It would cost time and resources, but so would customization of an external tool. The first production version of CmmnBuild was deployed in with a minimum of functionality.
Since then, it has been incrementally developed and new functionality has been added when needed. CmmnBuild is still under development as new requirements from the users appear.
As CmmnBuild evolved and was taken into use by all Java-developments, it was decided to rewrite the Release Tool to better suit the needs of the new language. Ant is specifically aimed at supporting Java. The Release Tool is still under development as new requirements from the users appear.
The build and release tools 31 5. Client-server design The relationship between CmmnBuild and Release Tool is based on a client-server model. The tools are, however, not tightly linked and CmmnBuild is just one of several clients using the Release Tool. Other clients are not considered here. Figure 5. A short introduction to the use of Ant in CmmnBuild and Release Tool is presented before elaborating on the technical aspects. Example of such tasks are javac, mkdir and echo [2].
In addition to these core tasks, there are several additional tasks available through external packages easily included into the local Ant installation. Examples being the ant-contrib [21] and the xmltask [20]. Even more specialized tasks can be written if needed. All underlying logic of Apache Ant is written in the Java language, and extra functionality may therefore easily be written in Java. Most of the underlying logic of the services performed through CmmnBuild and the Release Tool is provided by standard Ant packages or extensions such as xmltask and ant-contrib.
In addition, some specialized features are implemented in Java. These are further explained when needed. Technical details of CmmnBuild CmmnBuild consists of a set of ant-scripts divided into several files based on their functionality.
The main file is the common-build. Through these scripts CmmnBuild can provide its services through a set of predefined targets. CmmnBuild offers no graphical user interface to its users, and the targets may be run from command line or through an IDE.
This requires an Ant plugin. CmmnBuild includes three main areas of support: creating a distribution of a product, fetching dependencies and releasing the product. Not all targets are available for direct execution. The properties are explained later. Table 5. Targets Action usage prints all targets available for the user compile compiles all source code dist creates a complete distribution of the prod- uct getjars fetches the dependencies specified from the repository release releases the product into production; involves Release Tool devrelease releases the product to the Qfix folder junit executes all unit tests written for the project Table 5.
The build and release tools 33 The dist target is the main task of CmmnBuild. It performs all tasks which are needed to create a full distribution of a product. The resulting output depends on the properties specified for the project. CmmnBuild is a centralized tool, meaning that it prevents downloading a big buildfile for every new project. An important goal for CmmnBuild is to keep the overhead of using it as little as possible. In addition, the user must adapt a certain directory structure which includes four special configuration files the buildfile included.
Advanced users may extend this file with custom targets. It contains the name, the version, CVS module name and may also contain a description, a webpage link and a set of dependencies.
When specifying the dependencies in the product. The production version is explained later. An example is enabling the generation of javadoc by setting the property javadoc. The last file is the people file, indicating the names of the users who have the right to release the product. It is used during the release process figure 5. The focus of this report is on Java projects using CmmnBuild. In this case, Release Tool is not used directly by the user, but is called from CmmnBuild when running the ant release command.
Before CmmnBuild invokes the Release Tool, it tags the the collection of the artifacts making up the product in the CVS with a version tag. Production repository contains all released products of accelerator control software developed in-house. The 3rd party repository contains third party libraries used by developers in their projects. The 3rd party repository resembles the Gentoo Portage Tree in the way that it stores and maintains external software.
Example of 3rd party libraries are JUnit and Log4J. The release tool only supports releases to the production repository. Information about the content of the repositories is contained in xml files, referred to as repository. This is further explained later. When a product is released, the Release Tool extracts this product from the CVS to the dedicated production repository and builds the product. In case of Java products, the release will call its own local CmmnBuild installation to build and prepare the product.
To maintain the loose coupling between the Release Tool and CmmnBuild, a different build or make tool can easily be used instead. The directory structure corresponds to the structure in the CVS repository. For every new version, the Release Tool creates a separate directory, taking the version as name, without modifying old versions. This leaves the opportunity of going back to an older version at a later point.
As default, a product when released, becomes the production version, but there is the possibility to release a product as the next version pointed by the NEXT alias.
This feature may be used to evaluate a release in its real environment before releasing it as the production version. The PREV alias allows the user to easily revert to the previous version in case the newly released production version does not work as planned.
The motivation for this possibility is to supply the developers with a place to quickly release a bug fix or untested functionality. The Qfix folder has full write access which means that a developer can modify his code immediately, even without being at his usual workstation.
Products released to Qfix are not related to any version number and will be overwritten at next Qfix-release. The Qfix can be affiliated with any of the aliases. The repository. Both 3rd party and the production repository keep such a file mirroring their content.
These files are the source of information when interacting with the repositories. CmmnBuild uses them to resolve and fetch dependencies, and the Release Tool uses the repository.
In addition, these files are available, through the use of XSLT, for a user to view via a web-page. Actions are performed through ant commands. Consistency between the repositories and their respective xml files mirroring their content is crucial for the operation of CmmnBuild and Relaese. For the remainder of this report, when referring to the repository. The following chapter will summarize the functionality available, and introduce the areas in which new functionality is desired.
What services were already supported, and what were the areas in which new functionalities were desired. Furthermore this chapter presents the problem formulation for this thesis. What were the goals to achieve. The goal of each extension has been to develop a functional feature, and making this available for users in the newest versions of the tools. The tools have provided support spanning from the startup of a new project, or a new version, to the installation of that product in the production repository.
This included all steps in the development process presented in chapter 5. Table 6. Instead, a new version is released with the proper changes.
This might cause a problem if the version with the bug is old, and the project is in such a state that a bugfix cannot be provided any time soon in the newest release. The CVS provides a service called branching to allow a developer to create a parallel development path of a project based on a version tag [6]. Incorporating the branching functionality into CmmnBuild was the first problem addressed. With the startup of the LHC closing in, and the installation of the new control center, much software is produced and the amount of software kept in the repository increases continuously.
To keep the repository in an easily maintainable state, functionality to remove unnecessary products was wanted. It is used by developers and users to register issues about a product.
Such issues can consist of bug-reports, wanted functionality or a change log. Currently, this tool works independently of CmmnBuild and the Release Tool. An overall goal is to merge the tools in order to support the full product lifecycle. As a start, an automatic release notification scheme was looked into. These issues led to the following problem definition. This project represents three such increments responding to the problems presented in the previous section. The overall goals for the increments are shown in table 6.
The purpose of this thesis has been to implement support for each of the goals presented, and make this available in new versions of the tools. An elaboration on the issues and specific requirements for each goal are presented in chapter 8. Before the decision was made to develop an in-house solution for build and release management, several external tools were considered. One reason for the final decision was that the tools were found immature, unstable and lacking key functionality.
Today, however, many of these tools have grown to become widely used SCM tools, and the question has been asked whether it might be beneficiary for this section to switch one of these tools.
A final goal of this thesis is therefore: Based on the experience of the last 6 months improving the SCM solution, could it be beneficiary to migrate to an external tool to solve the sections SCM needs?
The goal is not to give a final answer to such a question, but to shed some light on the subject which might be of value in an upcoming discussion. Packaging to jar and war files is possible. In addition, gen- eration of javadoc, jnlp and html is available.
Some of these are available as stand-alone services. Some of these services rely on customized Java packages, while the rest are based on the functionality provided through Ant. One exception is code generation, which is a customized target relying on XSLT Dependency Provides the service of fetching the dependencies of a prod- management uct from either the production repository or the 3rd party CmmnBuild repository.
This happens transparent to the user. This service is made available through a customized Ant task developed in Java Release Builds and installs a product in the production repository. Also available is the generation of a command script Release Tool for launching the application Table 6.
Problem Elaboration Desired extensions to CmmnBuild and Release Tool Subject Goals Incorporation incorporation of the CVS branching feature into Cmmn- of branching Build facilitating the process of providing a bugfix Repository provide functionality to limit the amount of manual la- maintenance bor necessary to keep the repository-server in an easily maintainable state Release notifi- provide support for automatic release notification.
A cation first step in a more extensive solution Table 6. Presented in Part II is the current state of Software Configuration Management with emphasize on key functionality for build and release management. Figure 7.
The two parts are further explained in the following section. In addition, important features and functionalities of build and release management were explored through studying a set of external tools in addition to tools presented in this thesis. The search for articles was carried out on search-engines covering databases and on-line article repositories accessed through licenses of NTNU.
Research Method 7. Since the project consisted of developing independent extensions to the tools, it was natural to assume an iterative, incremental strategy on the overall level. The overall level referring to the development of each new feature.
There are several approaches to software development depending on the type of project, the size and location of the development team, customers etc. Three well known approaches are the waterfall model [22], iterative development [18] and agile methods [1]. The methods are often recognized by their abilities to adapt in changing environments. The waterfall method is the most predictive of the three, while agile methods are considered most adaptive.
A mixture of both may be found in iterative development. Agile methods emphasize real-time communication over written documents, with face-to-face communication as the preferred way. Development using agile methods consists of dividing work into small iterations, each iteration often representing a mini-increment of the software [30].
The development team has mainly consisted of two people, the author of this report, responsible for the implementation, and a project leader, responsible for the tools in total, and acting as an advisor on implementation issues.
Because of these factors, it was natural to adapt an agile development strategy based on face-to-face communication, both within the team and with customers. To be able to react fast in change of requirements, each extension implemented consisted in several iterations which were planned, implemented and tested along the development path. Included here is the implementation of the three new features of the build and release management solution presented in chapter 6.
Section 8. It was the first extension implemented. The branching feature does not represent a grand implementation and served therefore as a good introduction to the tool. As for all three implementations, scenarios and specific requirements are presented in their respective section. The most substantial extension implemented was the repository maintenance, section 8.
The feature was to be incorporated into the cern-jjar package, containing logic for interaction with the repositories. The package was already used to fetch dependencies and to update the descriptor file for the production repostitory during release. The functionality for updating of the descriptor file was rewritten during this implementation.
The old solution of this function is not an object of focus in the elaboration of the implementation. Last is the implementation of the release notification scheme. Automatic mail creation is a type of feature which is not always welcome, as it might lead to heavy traffic in the inbox of programmers involved in many projects. It is therefore not only important to offer developers means to choose only certain projects for subscription, but also provide them a way to stop the notifications.
Chapter 9 evaluates the process and the result of all three sections. Extending the build and release management solution 8. One of the features provided by the CVS is called branching. Branching provides the functionality of creating a parallel programming path, independent of the main branch figure 8. This feature of the CVS can easily be used manually through command line or an IDE in our case, Eclipse , but incorporation of this feature into CmmnBuild was desired, thereby making sure it is done in a consistent and standardized way.
Figure 8. A specific version is not maintained and updated in the repository. New versions may be released, and old versions may be deleted, but apart from that, a released version is static. This means that a version is never patched to suit the need of a user, even if it contains a bug. Instead, a new version is released.
A specific scenario for the use of the branching-feature is further explained in the following section. A concrete example follows. Branching 49 Fixing a bug in an old version Frank, responsible for a large low-level application, receives a call from an operator reporting a bug in Franks software. The operator himself was developing a new feature to a program, when he realized that there was a bug in one of his dependencies.
He tells Frank that he is depending on version 3. This version, which dates six months back, was the last version released by Frank before a major change of the application was initiated.
The feature desired by the operator was not available in the version before, and has not yet reappeared after the restructuring. To avoid going back on his new changes to fix this one bug, Frank creates a branch at the point the version which the operator is using.
By creating a branch, he will be able to fix this bug in an independent and parallel programming path. This way, he can provide the operator with a working product until he will release his new and improved version. To minimize the manual actions necessary, a checkout of the new branch should automatically be performed after the branch-creation.
Default values should be provided for both branch tag name and check-out folder, but users should also be allowed to specify this themselves. No default can be provided for the version number to branch. The following functional requirements were defined. If not, default values will be used.
The functionality is placed in the script which is also responsible for releasing a product. The branch command is made available as a direct target for the user. If no version is specified, a popup box will appear for the user to provide it. For the other properties, default values are used in case they are not specified. User executes the branch-command, and gets prompted for version to branch and password 2. CmmnBuild connects the user to release server using SSH 3.
The server tags the desired version in the CVS with a branch tag 4. Control is returned to CmmnBuild, which checks out the branch that was just created. The product is installed in the specified, or default, directory. Branching 51 necessary software on every machine using CmmnBuild.
The checkout is a read-operation and is executed from the local machine. Following is the implementation of the Branching. Maintaining the production repository 53 8. Whenever a release is executed, a new version is built and installed on this server in the correct place. The problem of any repository is the lack of infinite storage space, and every release is place consuming. More important, a repository growing uncontrolled is harder to maintain.
There has never been in place an automated process to keep the repository in a maintainable state by removing obsolete products. This has been done manually by the administrators of the Release Tool. The process includes both removing the physical files from the server, and updating the file mirroring the repository, repository.
As the xml file grows, the chance of human error during editing increases. In the current state of the CCC, new software is being developed and tested continuously. Table 8. The growth further is illustrated in figure 8.
This would reduce manual labor of keeping the number of artifacts from growing uncontrolled, and decrease the possibility of leaving the repository and the mirroring file in an inconsistent, unmaintainable state. The following definitions were put in place: Obsolete version - a version which is not depended on by any other products, and that is not tied to an alias.
Number of versions to keep - the number of obsolete products the user wants to keep from being removed from the repository The release of a product should only clean obsolete versions of that specific product. Obsolete versions of other products should remain unchanged.
The process, hereby referred to as AutoCleanUp resembles the functionality referred to as withdrawing a release in SRM [24]. Extending the build and release management solution Figure 8.
Cleaning all obsolete products Lisa is ready to release the newest version of her product. She wishes to release her newest version as production version, alias PRO, and she would also like some of the older version to be removed.
She considers her newest release quite stable and concludes that there is no point of keeping a long history of versions. Knowing that the default number of versions to keep is set to two, she overrides this property in her project. When releasing, all obsolete versions will be deleted, saving only the ones that are being depended on, or the ones affiliated with an alias. Saving obsolete products John is ready to release. He would like to release some new features of his product as a development version, thus with the alias NEXT.
Maintaining the production repository 55 he takes a look into the repository and sees that not all obsolete products should be removed. Two of the latest version numbers, not be depended on by any other project, and not containing an alias should not be deleted. Knowing that the default value of number of versions to keep is two, he does not have to reset this property in his project. During the release, two of the products considered obsolete will be kept. In order for CmmnBuild and Release Tool to work properly, it is crucial that the file system and the repository.
In case an obsolete product is deleted against the will of a developer, it is important to know exactly which versions were deleted. This will allow us to re-release the versions in question based on the source files kept in the CVS.
Keeping track of which files are deleted and when is therefore important. If for some reason, the automatic cleanup of the repository is no longer desired, it should be possible for an admin of the tools to switch off the removal of obsolete versions. In addition to the definitions presented earlier, this led to the following functional requirements. Provide a default value of 2 see definition in 8. Cleaning obsolete products is only supported for users releasing through CmmnBuild.
The logic for the AutoCleanUp is placed in the ant-script updateandclean. In addition to use a set of standard Ant targets, this service relies on the cern-jjar package to fulfill its requirements.
The cern-jjar package is written in Java and provides custom functionality to CmmnBuild and Release Tool. Its services are used from the ant-scripts. To accommodate the functionality needed for the AutoCleanUp, cern-jjar was extended with the ReleaseTask. This task handles the update of the repository. The list of obsolete versions is returned to the ant-script where the deleting is carried out.
An overall illustration of the process is provided in figure 8. Maintaining the production repository 57 8. It is divided into sections based on the sequence of actions making up the whole process. A detailed illustration of the sequence may be found in Appendix B. This file is parsed using the JJarParser which uses the Document Object Model, DOM [27], providing the possibility to change and update the information contained by the file.
The information about the new project being released is taken from the product. After parsing, this information is modified using XPath [28] to fit into the repository. The new product info is then added to the file, followed by an update of the aliases for the current product.
The total process of updating the repository descriptor is done in two steps. Extending the build and release management solution is added and the aliases are updated, and second, the obsolete versions are removed from the document. The document is not written to file between these actions.
Calculating obsolete versions The decision to only consider versions of the current product when calculating obsolete versions, limited the need for a complex algorithm. After the obsolete versions of the product have been calculated, the user-specified number of obsolete versions to save are removed from the list of versions to delete.
Ant provides a core-task for deleting files based on a relative path. The list of versions to delete is given to the ant-script as a comma-separated list of paths. It is provided by setting a property in ReleaseTask called product. Following is a section of the ant-script responsible for calling ReleaseTask and deleting the versions deemed obsolete.
Maintaining the production repository 61 Consistency For the release-server to function properly it is crucial that the repository. If for any reason, an exception should occur in the ReleaseTask, it must be possible to perform a roll-back leaving the situation as it was before the release was initiated.
If an exception is thrown, a proper message is displayed, and the release of the product will be rolled back, leaving both the repository and the repository. Logging A logging of all releases was already in place. All that needed to be done was to add a list of which versions of the product being released was deleted. Admin switch A property, switch. Default value is false, but if set to true, the cleanup feature will be deactivated for all releases.
This property is, as the name implies, only available for administrators of the tool. This includes several low-level applications relied upon by many other projects. Each time a product is re-released with changes in form of extensions or improvements, this might affect anybody depending on this project.
The section uses a JIRA which contains information about each project under development. The JIRA tool offers users the possibility to register issues in form of bug reports or requests, while it provides the developers the opportunity to update information of the current state of their product. This however, requires that the developer actually updates the information and that the person interested checks to see if there are any updates.
After the release scope projects, stories, and so on is finalized, you can generate Change Items and associate them to the release, allowing the implementation and deployment of a release to be handled within the change management process. Activate the Release Management plugin com. Use the business stakeholder role to read and retrieve data from any table of the Release Management application to generate reports.
This is an overview of domain separation and Release Management. Domain separation enables you to separate data, processes, and administrative tasks into logical groupings called domains. You can control several aspects of this separation, including which users can see and access data.
0コメント