A Comparative and Analytical Study for Choosing the Best Suited SDN Network Operating System for Cloud Data Center

Maiass Zaher and Sándor Molnár, “A Comparative and Analytical Study for Choosing the Best Suited SDN Network Operating System for Cloud Data Center”, Annals of Emerging Technologies in Computing (AETiC), Print ISSN: 2516-0281, Online ISSN: 2516029X, pp. 43-59, Vol. 6, No. 1, 1st January 2022, Published by International Association of Educators and Researchers (IAER), DOI: 10.33166/AETiC.2022.01.003, Available: http://aetic.theiaer.org/archive/v6/v6n1/p3.html. Research Article


Introduction
SDN paradigm introduces flexibility in data networks as network devices comply with NOS instructions by separating the control and data planes. In particular, switches in the data plane perform packet forwarding related functions as determined by the control plane. The so-called NOS runs on computer hardware [1], and the control plane represents the business logic in SDN paradigm as it is the controller of network functions. Specifically, it determines how to handle incoming data packets using protocols such as OpenFlow [2]. On the other hand, SDN paradigm also provides a network abstraction for applications at the management plane so that the operator can efficiently perform different network tasks. Figure 1 shows the layers of the SDN paradigm. The layers separation provides flexibility for introducing new solutions for problems of the traditional network paradigm. The layered architecture provided by SDN simplifies the deployment of network services and functions such as virtualization, packet forwarding, troubleshooting, etc. SDN layers communicate with each other using dedicated Application Programming Interface (API) as depicted in Figure 1. Therefore, adopting the suitable NOS has significant impacts.
CDC motivated to develop SDN paradigm as it relies on virtualization technologies to provide services within the cloud environment. In this context, the traditional network paradigm does not provide the required flexibility to apply these technologies. In addition, data centers have unique traffic patterns, which are different from those in the traditional networks [3]. As a result, CDC becomes one of the most implemented use case of SDN paradigm [4]. Therefore, many commercial companies, organizations and www.aetic.theiaer.org research centers have developed NOSs, but no specific NOS that can be used for all use cases because there are several alternatives according to the requirements. Hence, the research problem was embodied in answering the following questions: (i) Which NOS is the best suited for CDC? (ii) What are the functional and non-functional specifications to be verified for this use case? (iii) How important is each of the specifications to answer the first question? The importance of this research lies in the following aspects: 1) There are many NOSs, and their development frequency is rapid. Therefore, selection of the appropriate NOS for CDC requires an analytically comparative study.
2) The selection of the appropriate NOS for the aimed use case profoundly impacts the offered services as it is the control plane in SDN paradigm.
3) This study is expected to help researchers and operators in this field to transform from traditional network to SDN paradigm, identifying the most common open-source NOSs and determine the most appropriate NOS for CDC. This paper has the following contributions: a) Framework to determine the best suited NOS to be used in CDC. b) Provide an analytical study of the functional and non-functional specifications of six open-source NOSs.
The paper consists of the following sections. In Related works section, we present the literature review. In Research Methodology section, we overview the used research methodology. We introduce the problem and explain the employed decision-making method in the problem statement and AHP analysis section. We discuss the results in Results and discussion section. Finally, we present the conclusion in Conclusion section.

Related Works
Several research works compared NOSs. Khondoker et al. [5] conducted a comparison of specifications for each of the following NOSs, (ODL, Ryu, POX, Ryu, Floodlight, Trema). Ryu was found to be the best NOS according to the adopted criteria for the evaluation. Khondoker et al. did not consider so common NOSs such as ONOS and this study considered no specific use case. The study in [6] compared southbound API, Openflow version, programming language and round-trip time (RTT) delay of POX, ODL, ONOS, Ryu. They found that Ryu has the best score based on TOPSIS method. The work in [6] considered a small set of criteria which are OpenFlow version, NOS programming language, RTT, interfaces and documentation. However, we consider more criteria in this study to evaluate different NOS specifications. The work in [7] presented a feature AHP-based comparison of ODL, NOX, Beacon, Trema, POX, OpenMUL, Ryu, Floodlight, OpenContrail and ONOS in terms of traffic classification as the targeted use case. They found that ODL provides the best specifications for the use case. However, the study did not consider criteria related to cloud based data center similar to what we consider in Table 3. The work in [8] presented an ANP-based comparative study of features and performance of Floodlight, ONOS, ODL, POX, Ryu and Trema. The authors found ODL has the best score. However, this study considered many quantitative measurements such delay, throughput and CPU utilization to find the optimal NOS whereas we consider only the qualitative criteria in this study. The comparison conducted in [9] presented the optimum NOS for deploying SDN-WAN. The authors compared Ryu, ODL, POX, Trema, Floodlight.
www.aetic.theiaer.org Using AHP, the authors tackled the problem of finding the best NOS might be used for real use case which is connecting university campuses using SDN instead of Multiprotocol Label Switching (MPLS) Border Gateway Protocol (BGP) technology. Amiri et al. [10] compared features and performance of many open-source NOSs by using Best-Worst Method (BWM) decision-making method and Cbench to assess the qualitative and quantitative criteria of NOSs. The work in [10] employs Cbench to measure throughput and latency as quantitative criteria and ODL has the highest rank in both. However, the authors did not aim a specific use case. Ali at el. [11] proposed a prioritized features method used for selecting the best NOS. The authors employed ANP to implement the proposal. The investigated NOSs are ODL, ONOS, POX, RYU and Trema. The aim of this study was reducing the computational complexity of the selection process by selecting high weight features and ignoring low weight features. However, the study did not consider a wide spectrum of criteria and it did not aim CDC as a use case. The work in [12] investigated many features of ODL, ONOS, Floodlight, POX, RYU and Trema to find the NOS which has the best feature set. In particular, the authors' final target was creating a hierarchical cluster of the best NOS to improve performance such as delay, throughput, fault tolerance and scalability. This work is not tailored for specific use case but it aimed to improve the cluster performance, and it did not consider CDC related features. On the other hand, many studies reviewed SDN state of the art to investigate different aspects of SDN solutions and NOSs. In this context, our study introduces different aspects of NOSs as well but for identifying the best NOS for CDC. In this context, Abuarqoub [13] reviewed SDN NOSs in terms of scalability. On the other hand, Sarmiento et al. [14] surveyed SDN NOSs in terms of the challenges yielded from inter-site networking services. Studies in [15][16] observed SDN based load balancing techniques and [18] reviewed SDN NOSs according to NOS placement problem.
However, to the best of our knowledge, we consider a use case that has not been considered in the literature. In this study, we aimed to find the best suited NOS for CDC. In addition, our study covers the support of different features such as fault verification and troubleshooting, traffic security and packet forwarding techniques. We conducted a qualitative comparison only since there are many previously published papers on performance comparisons, such as in [6][7][8]18].

Research Methodology
We experimentally verified the NOS specifications to decide on criteria and alternatives weights. We installed NOSs, and many of their features and services were investigated. Furthermore, we investigated information provided on their official websites. Several criteria were imposed according to the aimed use case. Initially, the specifications required to be provided by the suitable NOS were identified, as shown in Table 1. Then AHP [19] was applied by which the studied specifications of each NOS were evaluated.

The Studied NOS
The most common open-source NOSs presented at the time of this study were studied, which are ONOS, POX, RYU, ODL,Floodlight and Tungsten. The reason for choosing these NOSs is that they are considered the most widespread open-source NOSs. Most of the other open-source NOSs are either 1 "Pox-doc", McCauley, last modified 2019, https://noxrepo.github.io/pox-doc/html/ 2 "ryu the network operating system(nos)", RYU Community, last modified 2019, https://ryu.readthedocs.io/en/latest/index.html 3 "Floodlight controller", Floodlight Community, last modified 2019, https://floodlight.atlassian.net/wiki/spaces/floodlightcontroller/ 4 "Start guide", Opendaylight, last modified 2019, https://docs.opendaylight.org/en/stable-sodium/getting-started-guide/index.html 5 "Onos", ONOSProject, last modified 2019, https://wiki.onosproject.org/display/ONOS/ONOS 6 "Tungsten fabric", Linux Foundation, last modified 2019, https://tungstenfabric.github.io/website/ www.aetic.theiaer.org designed for a specific purpose or no longer being developed. However, all unconsidered NOSs in this study are used less according to the frequency of their appearance in research works [20] [21]. ONOS is the most recent NOS produced. It is developed by ON-Lab, which is a non-profit organization. One of its main aspects is that it supports legacy networks. POX was developed to be the successor to NOX. an open research community developed POX. It is suitable for conducting SDN-based researches . According to the component-based model, Ryu has been developed by NTT company to facilitate the process of adding and modifying its components in any programming language. Big Switch developed Floodlight, and its main goal was to provide a NOS capable of dealing with open network hardware (i.e., hardware that does not contain software and is called "white box"). As a result of the effort of a consortium of several international companies, under Linux Foundation, ODL was introduced to obtain the acceptance of companies and users and to receive improvements continuously. Finally, some NOSs are considered as cloud oriented SDN NOSs where they aim at managing the network infrastructure of CDC. In this context, we consider Tungsten which is the open source version of Juniper Contrail and now it is a project of Linux Foundation. Table 1 lists versions of NOSs involved in this study.

Characteristics of Cloud Data Center
Nowadays, cloud computing drives most businesses and shapes a new era of technology delivery. Cloud computing provides different services such as software, storage, and virtual resources, so it abstracts the technical complexities, and it eliminates the cost and risk associated with hardware maintaining and acquisition [4]. In recent years, SDN has been employed in CDCs since SDN provides central control of the network. SDN-based Data Center Network (DCN) is preferred to the traditional DCN since it can improve DCN efficiency [22]. Furthermore, hypervisors are used in CDC to enable the deployment of virtual resources, so functions such as traffic control and isolation are crucial for providing efficient services. In this context, SDN and Network Function Virtualization (NFV) are seen as enablers of network slicing to create multiple virtual networks over a shared infrastructure. Each virtual network can be logically isolated from other virtual networks and assigned to serve specific requirements. Since SDN NOS has a global view of the network, traffic routing decisions can satisfy service demands, and each network slice can be provisioned with network and cloud resources based on QoS and Service Level Agreement (SLA). Due to the programmable model of SDN and the separation of the control plane and data plane, SDN facilitates the deployment of network functions that are essential for managing virtual resources where Virtualized Network Function (VNF) can be deployed in networks dynamically by SDN's configuration and automation functionalities.
In this context, many protocols are used as a southbound interface so that NOS can programmatically contact the switches to exchange information and send instructions. Besides, many NOSs expose northbound interfaces that can be used by the management plane and cloud orchestrator to implement different solutions (e.g., Quality of Service (QoS) management, traffic engineering, fault recovery, network statistics, topology discovery, etc.). Furthermore, the dynamically changing status of the network should be considered by the cloud systems in terms of the management of computing and storage resources, Virtual Machine (VM) provisioning, and addressing virtual resources requirements [23,24]. The massive scale of a CDC, which consists of thousands of compute nodes and network devices, imposes the necessity for network management, performance improvement and dynamic provisioning of computing and network resources [25]. Therefore, the architecture of the control plane should be taken into account whether it should be centralized or distributed. The centralized control plane represents a single point of failure. Although centralized architecture might be an efficient solution for small scale CDCs, it may not be efficient for large scale CDCs. Therefore, a distributed control plane consisted of many NOSs should be deployed. However, such architecture yields other challenges such as synchronization and heterogeneity of the underlying data plane [26]. On the other hand, SDN paradigm triggers additional motivations for the attackers. SDN NOSs and OpenFlow are considered potential security vulnerabilities which require efficient security solutions to improve the safety [27]. In particular, cloud orchestrator and SDN NOS integrate each other. Cloud orchestrator manages virtual resources like VMs provisioning, while SDN NOS manages physical and virtual network resources by southbound API (e.g. Openflow) and VM traffic. Besides, they communicate by north-bound API (e.g., Rest API) [28]. Hence, since SDN NOS and CDC www.aetic.theiaer.org orchestrator are integrated, maintaining the security is vital. Therefore, security solutions should utilize SDN capabilities and protect SDN components as well.  In this context, SDN can be employed to prevent a distributed denial-of-service (DDoS) attack by utilizing SDN functions such as traffic analysis. On the other hand, the SDN NOS must be secured itself as it represents the central control layer in SDN paradigm [29]. As Big Data and Internet of Things (IoT) applications exploit CDC, utilizing SDN-based CDC can enhance the performance of these applications by applying load balancing and QoS solutions [30]. For this sake, SDN can provide different QoS mechanisms for different applications, such as bandwidth slicing. In addition, scalability is an essential concern [31]. Since SDN NOS gathers information from data plane devices, it is crucial to avoid deploying a single SDN NOS in large scale CDC. Single SDN NOS represents a single point of failure, which could be problematic because there is a big number of devices in the data plane of CDC. Therefore, the distributed architecture of many SDN NOSs, to improve the availability, can be employed using westbound and eastbound APIs to maintain the synchronization [32] [33]. Furthermore, the faults represent a critical challenge to meet SLAs. In CDC, fault tolerance techniques provide the ability to maintain QoS. In this context, improving the reliability in CDC can be proactively achieved to avoid fault occurrence. For example, system load can be identified as an indication of potential future failures so that new resources might be added to avoid failures proactively [34] [35]. Although there are many opensource NOSs, and they are differently matured in terms of CDC requirements. It is challenging to find NOS that can provide CDC requirements, and it can integrate with the CDC orchestrator. As a result, based on the former characteristics of CDC, we summarize in Tables 1 and 2 the functional and nonfunctional features, respectively, which are required to be provided by the best suited NOS. The protocols used to manage the settings of network hardware by NOS Security The security features provided by NOS to perform its functions safely

Scalability
The ability of running multiple instances of NOS within a distributed and consistent cluster Availability The ability of providing services by NOS in case of failures www.aetic.theiaer.org

Decision support system
In this study, AHP [19] is used to make a decision according to multiple criteria. It is the most appropriate method according to the nature of the problem since it is a mathematical method for decisionmaking, which generates criteria weights through pair-wise comparisons. This comparison is also used to evaluate alternatives against each criterion. According to AHP, criteria and alternatives are treated separately. In addition, AHP presents problems in a hierarchical structure, which helps classify the criteria into levels according to the nature of the studied problem. Therefore, AHP reduces the size of the comparison matrices, thus maintaining the accuracy and consistency of results. Furthermore, AHP checks the consistency of the evaluation as long as the dimensions of the comparison matrices are under ten. On the other hand, other decision support systems like Technique for Order Preference by Similarity to Ideal Solution (TOPSIS) [36] is suitable for problems with negative criteria, while all criteria in this study have positive values only. Analytical Network Process (ANP) [37] is appropriate in case of correlated criteria and alternatives, while all the criteria and alternatives in this study are independent. On the other hand, Best-Worst Method (BWM) [38] identifies the worst and the best criteria in advanced so that the rest criteria are compared with them. However, in our case, we have many criteria, in different levels of the hierarchy, with similar importance what makes AHP the prefer method for this problem as well [39] [40].

The Problem Statement and AHP Analysis
This section presents the problem as a decision making problem applying AHP. The studied alternatives are where = 6, and the adopted criteria are where ∈ + . We aim to find the optimum NOS among the investigated NOSs using AHP according to the features listed in Table 2 and  Table 3. Therefore, the hierarchical presentation of the studied criteria is depicted in Figure 2-4. AHP requires to assign weights for the evaluation criteria. For this sake, we created × matrices for the criteria on the same levels of the hierarchy, and their elements are the priorities for each pair of the criteria to signify the importance of a single criterion to another, as shown in Eq. 1 where represents the importance of criterion in comparison to . In order to assign priority, AHP defines a scale between 1 and 9 to present the prioritization, as shown in Table 4 [19].
Then, the columns are normalized to find the relative weights by applying Eq. 2 Next, vectors of adding the elements of each row are created as shown in Eq. 3 By calculating the average of the previous values, we obtain a vector of weights Wm×i, which shows the weights of the criteria according to their pair-wise priorities, as shown in Eq. 4-5: The final weights of the leaf criteria are calculated by finding the resulted values in Eq. 5 multiplied with their parent criteria. The values of RI are presented in Table 5. According to [19], RI is the pre-calculated average consistency index computed by [19] of randomly generated comparison matrices with different scales, denoted as scale in Table 5. RI can be used as a reference value to check how CI, which is computed in Eq. 6, of our comparison matrices is far or close to the matched-scale RI. In particular, if CR, in Eq. 7, is 1, this means that we have completely random priorities. Therefore, we have no meaningful comparisons, and we have to revise our comparisons. The opposite is true in case Eq. 7 (i.e., CR) equals zero. In case CR is below 10%, the priority values are supposed to be consistent; otherwise, pair-wise priorities should be modified, and the pairwise comparisons should be repeated. Similarly, the alternatives are pair-wise compared against each criterion as in Eq. 8 based on the convention in Table 4, and weight vectors of the alternatives are computed, and their consistency is inspected as in Eq. 7, as well. Finally, the final value of each alternative is computed as in Eq. 9 where × is the weight vector of child criteria, ́× is the alternative's weights against all criteria, and × is a vector contains the final result of each alternative.   Tables 6-15 present the matrices resulted from Eq. 1. The pair-wise comparisons of criteria show that functional and non-functional criteria have equal importance. In relation to the non-functional requirements, we set weights of Ease of Use and Maturity less than that of the remaining criteria since all considered NOSs are open-source, and they can be developed according to the requirements. On the other hand, Development has a higher weight than Usability as NOSs might be modified to propose new functions and services. As a result, Modularity has a higher weight in comparison to Development Community and North API as it simplifies NOSs modification. As shown in Table 10, Documentation has a higher weight than GUI, where it is essential to adopt, use and develop NOSs. Finally, all remaining criteria have similar weights except Virtualization since we aim to find the optimal NOS for CDC. We present pair-wise comparison of the alternatives resulted from Eq. 8 in Table 16-33. Table 35 presents Ẃ × of all alternatives according to all leaf criteria. Final values of all alternatives, × , are presented in Figure 5.

Non-functional features
As shown in Figure 5, ODL is the best suited NOS for CDC according to the specified criteria. The final values of ONOS and ODL are close since they are being developed continuously by a large and active community, and they are supported by many industrial and research groups in comparison to the other NOSs. This presents the importance of adopting open-source software projects by relevant companies and institutions. Table 34 summarizes the detailed features of NOSs.

Scalability
ONOS, ODL and Tungsten are more scalable than the other NOSs, since they support distributed model in which multiple NOS instances can inter-operate simultaneously. However, ONOS can assign segments of Wide Area Network (WAN) or DCN to a specific instance to mitigate the instance load. ONOS employs Atomix database to store network information, which is also used for coordinating between the instances. In particular, ONOS represents a logically centralized control plane since the infrastructure might consist of multiple domains with one ONOS instance for each domain. On the other hand, ODL employs Advanced Message Queuing Protocol (AMQP) 7 to exchange information between ODL instances where each instance maintains information of its own domain. ODL leverages Federation and NetVirt projects to create virtual networks across many OpenStack instances, but these inter-site www.aetic.theiaer.org communications are managed at ODL levels, so overlapping is likely at Neutron level, e.g., duplicated IPs in different OpenStack instances is probable. Tungsten has control nodes run in a cluster to maintain scalability.

Security
All NOSs provide secure connections with the devices in the data plane by applying Transport Layer Security/Secure Sockets Layer (TLS)/(SSL). Moreover, ODL has Unified Secure Channel (USC), which consists of an agent, plugin and manager to initiate and maintain the connection with ODL, authenticates ODL and presents the state of USC in ODL DLUX user interface. Besides, ODL provides authentication, authorization, and accounting (AAA) framework controlling access to ODL features, applying policies to use those features and auditing the usage.

Easy to use & Maturity
In terms of Modularity, new modules can be added to all NOSs feasibly either as Python component or Apache Karaf. However, ONOS provides Yang management system that abstracts data modelling details of applications and device drivers by generating YANG modelled JAVA objects corresponding to device and application schema so that the business logic will be the sole concern. Hence, it can seamlessly support any interface like Representational state transfer (REST), Network Configuration Protocol (NETCONF), Extensible Markup Language (XML), etc. ONOS and ODL have the richest documentations. In addition, ODL and ONOS have the fastest update frequency and the most varied and numerous applications, but Tungsten provides richer GUI than other NOSs as configurations in terms of QoS, IP addressing, Ports, policies, etc can be depicted and defined using the web-based GUI. Finally, ONOS and ODL have the largest development communities as they are supported by many technological companies researching centers, and organizations.

Interoperability
Tungsten, ONOS and ODL provide Simple Mail Transfer Protocol (SNMP) as a management protocol which provides so many facilities in an environment such as CDC. In relation to south API control protocols, ODL provides Model-Driven Service Adaptation Layer (MD-SAL), which simplifies the communications between ODL modules and data plane through the available southbound APIs by supporting user-defined payload formats, including payload serialization and adaptation. ODL provides OpFlex 8 , which is an extensible policy protocol designed to exchange abstract policy between ODL and the devices support policy rendering. On the other hand, Tungsten provides Extensible Messaging and Presence Protocol (XMPP) 9 as a southbound API to communicate with vRouter 10 , which can be deployed either as a kernel module, Data Plane Development Kit (DPDK) 11 in user space, SmartNIC programmable network interface 12 , or by using Single Root I/O Virtualization (SR-IOV) for accessing Network Interface Card (NIC) from VMs directly.

Fault verification and troubleshooting
Most outstanding troubleshooting capabilities have been found in ODL and ONOS as well. ODL provides a diagnostics framework to report the status of ODL itself by performing continuous monitoring of registered modules and built-in services to maintain the overall health of the system, and it additionally triggers alarms. On the other hand, ONOS provides a framework to break the routing loop, routing black holes and applications conflicts. As well, ONOS integrates with services like CPMan 14 and Ganglia 15 to maintain the information of the ONOS state. In addition, ONOS provides fault management by polling SNMP capable network devices to track events like device adding, updating, link down, etc. Users can access and manipulate alarms by CLI, Rest API or GUI. FlowScale 16 provided by Floodlight diverts all traffic directed to a down port to other up ports. Tungsten's analytic nodes collect metrics from compute, storage and network nodes as well as their workloads to facilitate troubleshooting and monitoring. In particular, Zookeeper 17 is used to distribute the responsibility of collecting data among analytic nodes to avoid nodes overwhelming.

Packet forwarding technique
For the sake of improving the availability and the performance, ONOS balances the load between the instances in a cluster. Tungsten balances the load among the virtual resources by providing load balancing as a service employing different drivers such as HAproxy 18 . On the other hand, ONOS improves QoS by providing SDNi, which manages flow setup by considering information such as path requirement, QoS and Service Level Agreement (SLA). Besides, ONOS provides intent framework by which applications can specify their network control desires as a policy (e.g., tunnel provisioning, flow rule installation). Hence, different packet forwarding techniques can be applied at run-time. However, the other NOSs support QoS by creating queues, Openflow meters, Differentiated Services (DiffServ) or creating Label-Switched Path (LSP) by Path Computation Element Configuration Protocol (PCEP).

Virtualization
In relation to virtualization, all NOSs except POX integrate with Openstack orchestrator, support Virtual Local Area Network (VLAN), moreover, Tungsten, ODL and ONOS support Virtual Extensible LAN (VxLAN) and L3 tunnelling. In this context, ODL manages overlay tunnels established within a transport zone using VxLAN, Generic Routing Encapsulation (GRE), Generic Protocol Extension for VxLAN (VxLAN-GPE) and MPLS over GRE (MPLSoGRE). Besides, ODL provides L2, L3 tunneling, network address translation (NAT) and access control list (ACL) services for the virtualized resources using Neutron northbound API, which communicates with ODL driver in Openstack. On the other hand, ONOS provides a tenant network virtualization service for CDC provisioning using L2 VxLAN or L3 GRE. Furthermore, ONOS provides L2, L3 tunneling and NAT services using Open Virtual Switch (OVS) 19 in the compute nodes and provides horizontal scalability of the gateway node, which connects the virtual networks with the outside by BGP. ONOS isolates the virtual network traffic by Ethernet Virtual Connections (EVC), which uses VLAN ID to tag traffic of different EVCs uniquely and define the associated components like User Network Interface (UNI) and bandwidth profile. Similarly, ODL provides UNI manager, which provisions EVC in physical and virtual network elements of multi-vendor using YANG based APIs and drivers.             iconsistency ratio = 5.51%

Traffic protection solutions
Finally, Floodlight provides proactive ACL and basic firewall by installing Openflow flow entries reactively in terms of secure traffic. Besides, POX can block connections based on Media Access Control (MAC) address. RYU can apply firewall principles based on VLAN and Switch id. However, Tungsten, ONOS and ODL provide Service Function Chaining (SFC) by which an ordered list of network services (e.g. firewalls, intrusion detection system, etc.) can be defined, consequently deploying softwarized security functions in CDC becomes more flexible. In this context, OpenStack provides integrating SFC drivers to implement SFC using ODL and ONOS. Moreover, ODL provides a logical service function forwarder that facilitates service function mobility, load balancing and failover. In addition, ONOS provides a policy framework as an abstraction layer that hides the details of the control and data planes. This framework simplifies the enforcement of actions to the network by installing OpenFlow rules to provide functionalities like firewall and NAT. ODL provides NetVirt to apply ACL in ingress and egress modes to VMs created by Neutron northbound API. Tungsten provides security policies based on tags applied to projects, networks, vRouters, VMs and interfaces. Consequently, more granular policies can be created as well as security policy administration and troubleshooting are more feasible.

Conclusion
In this study, we presented the best suited NOS to be used in CDC by assessing the specifications of six NOSs, according to the criteria imposed by cloud data center requirements. Besides, we presented to which extend each NOS meets these criteria. To the best of our knowledge, this is the first study in the literature tackling such a problem. For this sake, first, we determined the requirements of CDC by investigating the characteristics of such an environment. Then, we classified the CDC requirements into functional and non-functional criteria based on which we inspected the specifications of the investigated NOSs. AHP was used, which requires much input conducted by pair-wise comparisons of the criteria to determine their importance by calculating their weights. As well as, we pair-wise compared the alternatives to determine the best one accurately against each criterion. Therefore, AHP's flaw is that it imposes many comparisons. We found that ODL and ONOS have the higher scores in comparison to the other NOSs. In particular, ODL has 31.29%, ONOS has 30.98% whereas Tungsten, Floodlight, Ryu and POX have 15.66%, 8.99%, 7.67% and 5.42%, respectively. A continuous follow-up of the NOSs should be conducted since Software Defined Cloud Computing (SDCC) is still in the stages of maturity and www.aetic.theiaer.org adoption. Moreover, the related technologies are developing rapidly because of the tendency to convert CDC into Software Defined Environment (SDE). In this work, we also presented the essential requirements for SDN based CDC. However, several requirements need more investigation under other use cases. In this context, it is essential to extend the current NOSs to provide network services in case of inter-CDCs, e.g., deploying SFC whose VMs belong to multiple CDCs managed by different cloud orchestrating instances. Furthermore, maintaining the synchronization between multiple NOSs belong to different CDCs managed by different cloud orchestrating instances is still problematic, and it needs more investigation to resolve potential synchronization conflicts. In addition, considering container based cloud environment sheds the light on new challenges in relation to the integration between the container orchestrator and SDN NOS. The previous challenges will be in our future work plans, where we need to investigate the NOSs capabilities and deficiencies to propose new solutions addressing the mentioned challenges.