Human Competency Assessment for Software Configuration Management

Syahrul Fahmy, Aziz Deraman, Jamaiah Yahaya and Abdul Razak Hamdan, “Human Competency Assessment for Software Configuration Management”, Annals of Emerging Technologies in Computing (AETiC), Print ISSN: 2516-0281, Online ISSN: 2516029X, pp. 69-78, Vol. 5, No. 5, 20th March 2020, Published by International Association of Educators and Researchers (IAER), DOI: 10.33166/AETiC.2021.05.008, Available: http://aetic.theiaer.org/archive/v5/v5n5/p8.html. Research Article


Introduction
Late product delivery is a common problem in software projects. Late delivery is caused by, amongst other things, software changes, leading to more development work than originally planned, thus increasing project completion time [1]. Software Configuration Management (SCM) has been used in software engineering to ensure timely completion of software projects and delivery of software products since the 1970s. SCM is a systematic change management process and can be defined as a "supporting process in software engineering that controls changes to software products" [2]. However, after more than half a century, the problem of late software product delivery and projects still prevail in software engineering [3].
One significant factor that influences the success of software projects is the human factor [4]. Although human factors have been studied actively [5][6], the focus are mainly on technological or process-related factors [6]. This is evident in SCM where studies involving human factors have focused on the process, for example collaboration and debugging [7], rather than the success of SCM implementation. Software development is a human activity [8], and human factors play a critical role in its success [9]. Competency is one of the many aspects of human factors and the lack of human competency has been identified as on of the issues in software development [10]. www.aetic.theiaer.org This study aims to identify how the competency of SCM practitioners can be assessed. There are two challenges in meeting this aim. The first challenge is to identify a set of SCM competency criteria as there is yet a standard set of competency criteria for software practitioners. The second challenge is to formulate a competency assessment framework for SCM. Major competency frameworks focus more on other software engineering areas and not explicitly to SCM. Both challenges are addressed in this paper.

Software Configuration Management
Software Configuration Management (SCM) is commonly used to manage the evolution of software systems. It is defined as "a supporting-software life cycle process that benefits project management, development and maintenance activities" [2]. SCM can be traced back to 1950s when poorly documented engineering changes in the aerospace industry posed a problem to spacecraft production. When Software Managers faced the problem of managing software change in the 1970s, SCM was adapted in the software development process [11].
SCM research and development (R&D) activities has been highly focused on addressing the problems at hand in software development for example programming in the large in the 1980s [12]; development of object-oriented systems in the 1990s [13]; constructions of web services in the 2000s [14]; and configurations of late binding systems post 2010 [15]. The corpus of research has been in the development of tools for software engineering. This has translated into wealth of SCM-related tools, making SCM implementation highly dependent on these tools. To date, SCM tools are pervasive and available to other areas such as web services and programming environments.
Although the role of human in software development has been acknowledged as early as the 1970s [16], R&D activities related to human factor in SCM have focused on the process for example collaboration [17]; education [18]; and debugging [19], rather than the success of SCM implementation. One aspect of human factor is competency. Competency is a set of characteristics that improve the efficiency and performance of a practitioner. The lack of (human) competency has been identified as one of the challenges in software development [10]. The quality of software products is highly dependent on the knowledge, abilities, and talent of the project team [20]. Traditionally, the competency of software practitioners was built on personal qualities and developed through experience. To date, formal education and professional training have become more significant in the development of competency.
Competency has many facets including knowledge, skills, abilities and personal attributes. It can be divided into specific [21] and generic [22] competencies. Unfortunately, there is no one size fits all or silver bullet for determining the competencies for software practitioners as organizational functions require a combination of competencies in order to be performed effectively [23]. In addition, there is no widely accepted competency framework for systems engineering. Major competency frameworks including PCMM [24]; INCOSE [25], MITRE [26], and SwA [27] are not suitable for this study as they focus more on other software engineering areas than SCM.
PCMM adapts the maturity framework of the Capability Maturity Model which aims to improve software development processes. As the scope of this study is SCM based on IEEE classification, PCMM is not suitable for assessing the competency of practitioners as it falls under the areas of Software Process and Software Construction. INCOSE presents description and indicators of knowledge and experience for competency in four levels namely Awareness, Supervised Practitioner, Practitioner, and Expert. However, there is no specific mention of SCM in the framework although the competency for Planning, Monitoring and Controlling is listed in the System Engineering Management category. This reduces the suitability of this framework for this study. In MITRE, SCM is discussed in the Systems Engineering Planning and Management section although no specific mention for tools utilization is made in the competency levels. The only mention of tools appear in the Intermediate Behaviour with the "recommendation of configuration management and deficiency reporting tools". As SCM implementation is highly dependent on tools, the lack of competency for tools utilization reduces the suitability of MITRE for this study. SwA falls under the Software Quality area in the IEEE classification of software engineering key areas and not SCM, thus reducing its suitability for this study. Although www.aetic.theiaer.org the Software Engineering Competency Model (SWECOM) [28] explicitly focuses on SCM, tasks remain in identifying suitable competency level for each skill set, and mapping specific skill set to a particular SCM process.

Identification of Competency Criteria
SCM competency criteria was identified in two surveys.

Questionnaire
The first survey identified SCM practice in software organizations throughout Malaysia. Software practitioners from the public sector, industry and Institutes of Higher Learning (IHLs) were selected as respondents. Questionnaire was selected as the survey tool as respondents are geographically scattered. The questionnaire was based on the standard SCM process [2] and the content was validated by a group of experts representing the target respondents. Construct validity was not carried out as the questionnaire was based on a standard document [29][30].
The questionnaire had 40 questions in 6 sections, and a total of 19 practitioners took part in the survey; 3 from the public sector, 11 from the industry, and 5 from IHLs. Results of the survey revealed 4 main SCM components namely People, Process, Tools, Documentation and their relationships as viewed by practitioners. Successful SCM implementation is dependent on the competency People to implement Process, operate Tools and generate Documentation. SCM implementation involves 6 interrelated process (Planning, Identification, Control, Accounting, Auditing, Delivery). The use of SCM tools are prevalent in software projects, as well as general tools for project reporting, progress assessment and project monitoring purposes. In addition, 3 types of SCM tools that are commonly used are software versioning, software building, and software release tools. The types of documents generated in software projects are in line with international standards. Minor variations are noted due to organizational culture and constraints.

Interview Sessions
Based on the questionnaire results, it is discovered that IHLs have the highest number of late project delivery, requiring an additional of 2.6 months for completion (on average). As such, the second survey specifically targeted IHLs to identify key factors that inhibit and promote SCM implementation. Open-ended [31] and unstructured [32] interview questions were used in order to get as much insights to significant areas in SCM. A total of 12 respondents from 5 IHLs participated in this survey in 9 interview sessions comprising of individual and group sessions, carried out at the respondents' organization. Results of the survey were summarized into SCM implementation issues and success factors.
People-related issues reported include conformance to directives, ineffective communication, encumbering key performance indicators, management's commitment towards timely completion of software projects, procedures for change request, bureaucracy, and additional task assignments. Process-related issues reported include ineffective communication, ambiguous requirements, inefficient change request procedure, manual change request process, frequent restructuring exercise, lack of infrastructure support for implementation, and lack of dedicated SCM manager to oversee implementation. Issues related to Tools include the adoption of technological change into the work process, tools suitability, tools utilization, diversity of target platforms for testing, implementation, and maintenance purposes, and vendor support. Issues related to Documentation include ineffective communication, poor documentation, lack of a dedicated SCM team, lack of understanding of SCM process and outcomes, lack of management's commitment, and vague organization directions.
Major contributing factors include a dedicated SCM manager to oversee implementation, efficient communication between all stakeholders involved, conformance to directives, training, clear organization direction, and management's commitment towards SCM process and implementation. Other contributing factors reported were the foundation of a Change Control Board, appropriate tools to support implementation, the use of SCM external consultant, the need for a suitable infrastructure www.aetic.theiaer.org to be in place, availability of SCM tools, SCM awareness, dedicated development and maintenance teams, clear requirements to minimize the number of change requests, and high morale of staff.

Competency Formulation
Filtering the survey results and leaving out non SCM factors such as budget and team size; and combining similar factors such as tools suitability and utilization; 10 significant factors to SCM were derived, illustrated in Table 1.
These factors are represented by a set of 5 competency criteria namely Knowledge, Experience, Professionalism, Training, and SCM Skills. These criteria are key for addressing SCM issues and promoting successful implementation.
The Malaysian Qualifications Framework [33] and the Malaysian Program Standards for Computing [34] were referred to in defining and refining the competency for Knowledge. Knowledge refers to the level of formal education in Computing and represented by education level and expertise in software engineering key areas. Education level refers to qualification obtained; area of expertise refers to the field of qualification; and key areas refer to the 13 software engineering key areas. SWECOM and SWEBOK [2] were referred to in defining the competency for Experience. Experience refers to the mastery of key areas through involvement in software development projects. The Software Engineering Code of Ethics and Professional Practice [35] was referred to in defining the competency for Professionalism. It refers to the professional conduct of a practitioner and has been adopted by the IEEE Computer Society and ACM. Training refers to the completion of workshops, seminars, and other learning mediums in software engineering key areas. SWECOM was referred to in defining the competency for Skills. It refers to the ability to implement SCM namely Planning, Conducting and Managing.

Competency Assessment Framework
SCM competency assessment evaluates the competency of practitioners in implementing SCM process, utilizing tools and generating documentation. Assessment also illustrates the organization's project management practice and highlights areas of concern. The assessment framework is illustrated in Figure 1.
Competency assessment can be carried out to all type software projects (new, current, completed, outsourced). Assessment for new software project would facilitate planning activities such as tools procurements and training needs. Assessment for current project is prompted by discrepancies in the project's schedule, amongst other things. As such, assessment is carried out to gain insights as to why the discrepancies occur and highlight areas for improvements. Assessment for completed projects is mainly a review exercise to determine what worked and what did not with regards to Human, Process, Documentation and Tools. Relevant data from past projects are collected www.aetic.theiaer.org and the results are used to plan future projects more effectively. Assessment for outsourced projects provides alternative input in the vendor selection process. Additional data can be included in the assessment, tailored to the requirements and needs of the organization. Competency assessment can be carried out through self-estimation surveys, interviews or work audit. In this study, a selfestimation questionnaire was developed and administered.

Human
Competency is determined based on academic qualification, professional experience, professional conduct, training, and SCM skills. Five levels of competency are adopted based on SWECOM and Dreyfus model of acquisition [36] namely Software Engineer, Technical Leader, Practitioner, Entry-Level Practitioner, and Technician. Determination of competency is as follows: • Software Engineer holds at least a first degree in Computing, has the proficiency of an Expert, with at least 5 years of Experience. • Technical Leader holds at least a first degree in Computing, has the proficiency of Proficient, with at least 5 years of Experience. • Practitioner holds at least and a first degree in Computing, has the proficiency of Competent, with at least 5 years of Experience. • Entry Level Practitioner holds at least a first degree in Computing, has the proficiency of a Beginner, with less than 5 years of Experience. • Technician holds at least a diploma in Computing, has the proficiency of a Novice, with any years of Experience.

Process
Competency is determined based on the extensiveness of SCM process implementation by the organization. There are 6 SCM-related process based on the IEEE-828 standard (Planning, Identification, Control, Accounting, Auditing, Delivery) [37]. Determination of competency is as follows: • Minimum competency for managing the Planning process is a Practitioner, as a practitioner "participates in determining impact of constraints imposed by policies, contract, and SDLC". • Minimum competency for managing the Identification process is a Practitioner, as a practitioner "participates in identifying CIs and the relationships among them" and "participates in developing software release plans". • Minimum competency for managing the Control process is a Technical Leader, as a technical leader "appoints members and convenes the Change Control Board" and "tailors and adopts mechanisms for requesting, evaluating, and approving changes". • Minimum competency for managing the Accounting process is a Technical Leader, as a technical leader "leads the Change Control Board in making decisions on change requests" and "ensures that approved changes are made and documented". • Minimum competency for managing the Audit process is a Technical Leader, as a technical leader "establishes and maintains the mechanisms for generating audit reports". • Minimum competency for carrying out the Delivery process is a Practitioner, as a practitioner "leads the building and verifying of software releases".

Tools
Competency is determined based on the tools used in the SCM project. Three level of competency is adopted based on SWECOM namely the specification, acquisition and utilization of tools. Determination of competency is as follows: • Minimum competency for specifying tools is a Technical Leader, as a technical leader "specifies SCM tools" and "establishes and maintains the mechanisms for generating audit reports". • Minimum competency for acquiring SCM tools is a Practitioner, as a practitioner "procures SCM tools" and "develops and configures tools for generating audit reports". • Minimum competency for utilizing SCM tools is a Technician, as a technician "operates SCM tools", "operates tools to generate status and audit reports", and "operates tools to build software releases".

Documentation
Competency is determined based on the documents generated in SCM. Two levels of competency are adopted based on SWECOM namely generating and updating documents. Determination of competency is as follows: • Minimum competency for generating SCM documents is a Practitioner, as a practitioner "develops and maintains the SCMP", "uses established procedures for populating and maintaining the library", and "uses established mechanism to record and report information". • Minimum competency for updating SCM documents is an Entry Level Practitioner, as an entry level practitioner "assists in developing, updating and maintaining the SCM Plan", "implement and document approved changes to CIs", and "provides audit reports as scheduled and requested".
To complement the assessment results and facilitate software organizations in taking corrective actions, several areas of concern can be derived such as training and professionalism. These informations would serve as a starting point to identify why a certain process, tool, or document is not implemented, utilized or generated although the minimum competency is met.

Validation
The proposed SCM competency assessment framework was validated through expert reviews. The identification and selection of experts were based on the industry. A total of 11 experts was selected in this study: 2 from the public sector, 3 from the industry, 4 from IHLs and 2 international experts. List of experts is as follows: • Semi-structured interviews sessions followed by a short questionnaire were administered to the experts, covering SCM components, competency criteria, and benefits of competency assessment to organizations. The interview questions were developed based on a brainstorming session with Computing post-graduate students at Universiti Malaysia Terengganu. The questions were then drafted into questionnaire and sent to a group of 5 computing lecturers (experts) at University College TATI. The final set of questions had 16 items, covering SCM components, competency criteria, and benefits of competency assessment to organizations. A 5-point Likert scale was used to record the responses. Validation revealed the soundness of the SCM components (Human, Process, Tools, Documentation); the competency criteria (Knowledge, Experience, Proficiency, Training, Professionalism); and the feasibility of competency assessment in different software development environments (new, existing, completed, outsourced). www.aetic.theiaer.org

Conclusion
This study started-off with the aim to identify how the competency of SCM practitioners can be assessed, leading to the identification of SCM competency criteria and the development of an SCM competency assessment framework. This study has successfully identified a set of SCM-specific competency criteria based on two surveys carried out and mapped against standard documents including the Malaysian Qualifications Framework, the Program Standards for Computing and the ACM/ IEEE Software Engineering Code of Ethics and Professional Practice. The selection of competency criteria was validated by experts from the public sector, industry, institutes of higher learning and international organizations.
A comprehensive SCM competency assessment framework has also been developed to determine the competency of practitioners, the extensiveness of SCM process implementation, the utilization of support tools, and the comprehensiveness of the SCM plan. The framework is based on international standards including SWECOM and SWEBOK. The framework was validated by experts from the public sector, industry, institutes of higher learning and international organizations.
This work has shed light to the roles played by human in SCM, complementing similar works such as the effect of knowledge to software quality [7][8], and contributed to the areas of SCM model and framework, where similar works include the impact of change to software quality and development [1] and component-based SCM model [38].