Analysis on testing and operational reliability of software

Journal of Harbin Instiute of Technology (New Series), Vol. 15, No. 3, 2008Analysis on testing and operational reliability of softwareZHAO Jing'2, LIU Hong-weir' , CUI Gang , WANG Hui aqiang'赵靖,刘宏伟，崔刚，王慧强(1. Dept. of Computer,Harbin Engineering University, Harbin 150001 ,China , E -mail: zhaoj@ hrbeu. edu. en;I2. School of Computer Science and Technology, Harbin Intitute of Technology，Harbin 150001 ,China)Abstract: Software reliability was estimated based on NHPP software reliability growth models. Testing reliabil-ity and operational reliability may be esentially dfferent. On the basis of analyzing simiaities and dfferencesof the testing phase and the operational phase, using the concept of operational reliabilityt and the testing relia-bility, different forms of the comparison between the operational failure ratio and the predicted testing failure ra-tio were conducted, and the mathematical discussion and analysis were performned in detail. Finally, sofwareoptimal release was studied using software failure data. The results show that two kinds of conclusions can bederived by applying this method, one conclusion is to continue testing to meet the required reliability level ofusers, and the other is that testing stops when the required operational reliability is met, thus the testing co8tcan be reduced.Key words: sofware reliability; non-homnogeneous Poisson process; testing reliability; operational reliabilityCLC number: TP273Document code: AArticle ID: 1005-91 13(2008 )03-0345-06The software system is the nerve of a computeroperational profile truly, accordingly, the fault detec-system. After fnishing developing software , the qualitytion rate( FDR) of the operational phase and the testingrequirement of the software is the key deciding whether phase of the software are different'3-7I , and the opera-the software can be put into practice, and one of the tional reliability predicted using the data of the testingimportant characteristics of the software quality is thephase is different from the actual operational reliability.software reliability. Computers are increasingly used in In actual application, it is very important to decidemany Safety Critical Systems', such as aviation anwhen to release the software.astronavigation，national defense and traffic. Thesesystems are called scs because their failures will cause 1 Differences of Testing Profile and Operationala great loss of lives and properties, therefore, the eval-Profileuation of the software reliability is especially important.The software reliability growth model can be used to es-An operational profile is a description of the distri-timate quantities associated with the software reliabili-bution of input events that is expected to occur in actualty, such as the number of remaining failures and thesoftware operation. During software testing phase, afailure ratio, and so on[2]. A good software reliability program is excuted with some inputs to see if the soft-model can not only predict whether the released soft- ware operates as it is specified. It is impossible to ex-ware is reliable in the operational phase, but provide a haustively test a program due to the sheer size of the in-wonderful gist for the software cost model. The testing put space. Thus some approach must be used to selectand operational environments of the software are as- a small subset of the input space with the hope that thesumed to be similar in most of the software reliabilityinputs from this subset are representatives for the wholemodels, in which the modeling on the failure process of input space and will be able to detect most of the soft-a software syslem is performed using the software failure ware faults. A plurality of testing methods, such asinformation obtained by testing, the reliability of the the boundary value testing, the equivalence class tes-software system is evaluated, and the field behaviors ting, and the path-based lesting considering the cover-when the software actually runs are predicted. In fact,age ratio of the testing are used, these testing methodsthe software performance depends on its operational en- will accelerate the failure processof the sofwarel9l.vironment which includes the operation system, the During operational phase , software testing is random.hardware platform and the operational profile. It is dif- In this anproach. test inout is selected randomly fromficult for the testing profile of a software to reflect the the in中国煤化工it will be fixed andReceived 2006 -08 -31.HCNMHGSponsored by the PhD Programs Foundation for Young Researchers of Ministry of Education of China ( Grant No. 20070217051) and Major Program of Na-tional Natural Science Foundation of China ( Grant No. 90718003 ).●345●Journal of Harbin Instiute of Technology (New Series), Vol. 15, No. 3, 2008hence the reliability is improved. The reliability growthdicted failure ratio using software reliability growthexhibited during software testing depends significantlymodel is hr, some scholars assumed that入。= λpon the selected inputs. Because of influence of acceler-It can been seen from the. above. discussion, it is notation testing of testing phase , the distribution of inputaccurate in the actual software testing that some schol-events of testing phase and operational phase is differ-ars assumed when discussing the comparison betweenthe testing reliability and the operational reliabilityl10].Because of differences of testing profile and opera-As for maintenance system, analysis of failure ra-tional profile, so that it is necessary to distinguish thetio of operational phase is similar to that of non-mainte-concept of testing and operational reliability. Assumenance system. So that for convenience, non-mainte-that the failure ratio of testing phase is decreasing, ifnance system is considered with regard to comparisonsthe software has been tested for sunits of teime andof testing and opertional reliability. The relation be-then released to customers. There are two kinds of ca-tween the failure ratio of the testing phase and that ofses about the varying trends of failure ratio.the operational phase is shown in Fig. 2.(1) As for maintainable system or fault deletionallowable of software beta testing, the varying trends ofλ()failure ratio of software testing phase and operationalphase is ilustrated as Fig. 1(a), where λ。(t) and入(t) represents separately varying trends of operation-|cal failure rate.Ir t(2) As for non-maintainable system, during the入二二七一S一十r-=opertional phase, fault removal is not the case and theuser will experience a constant failure occurrence rateover time. The varying cases of failure ratio of softwares+testing phase and opertional phase is ilustrated asshown in Fig. 1(b), where 入。,λ。represents separatelyFig.2 The testing reliability and the operational reliabilityfailure rate of operational phase，and λr represents thewhen the faiture ratio is decreasingfailure ratio at the time when the software is released.1.1 The Testing ReliabilityDefinition of the reliability (": the probabilitythat the software does not cause system failure in a pre-scribed time under the prescribed conditions. Accord-ing to the definition, the reliability of the system in the令|task time[t, t +x] is:R(x1t) = Pr| there is no failures during [1,t +x)}This parameter is applicable to systems that fail-ures are especially undesirable, such as a process control system and an aviation electronic system(a) Maintenace sytcemIn the testing phase, the testing reliability is in-creasing due to the continuous elimination of failures.If the elapse of time [s, s +x) is in the testing phase ,and the failure process of the software follows theNHPP process, the accumulated number of failures un--如til time t is expressed by N(t). Thus, the reliability ofthe testing phase [s, s + x) can be expresses as:R[(xIs)=Pr{N(s+x)-N(s)=0}=.exp{- [m(s+x) -m(s)]}.If the testing reliability is expresses by the failureratio shown in Fig. 2, the reliability of the testing phase(b) Non-maintenance system[s, s +x) can be expresses as:Fig. 1 Time-varying failure intensity from testing to opera-中国煤化工x) -m(s)]} =As for non- maintenace system, assume that theYHCNMHGWhere SABCD represents the area surrounded by theactual failure ratio of operational phase is入。, the pre-cure of failure ratio and the time [s,s +x)..●346●Journal o[ Harbin Instinue of Technology (Ner Series)， Vol. 15, No.3, 20081.2 The Operational Reliabilityanyt, Rp(x| T) > R_(x1 T).If the software is released to users after testing sThe demonstrations of Theorems 2 and 3 are sameunits, the operational reliability is calculated in theas that of Theorem 1.lapse of time[s, s +x) and can be expresses as:Theorem4 For anyT≥0 > 0, whenλ(t) isR(x|s) =exp(-广" 'λ.d)= exp[(- \。)x]firstly increasing and then decreasing, assume T。repre-sents the inflection point of the mean function m(t),Whenλ。= λr =入(s), the operation reliabilitythencan be written as:1) IfT。≤T, the discussion of the relation be-Rp(xI s) = exp[(-λ)x] = exp[ - SAacro].tweenRg(xI T) and R_(xI T) is consistent with thosewhere SABCD represents the rectangular area surroundedof Theorems 1 -3.by λτ and the time [s,s + x). It can been seen from2) IfT。 > T,3T so that入(T) =入(T),T >Fig.2 thatT, the discussion is as fllws:SAcD > SAacDa) IfT, > T+x, R.(xI T) > R.(x| T)Here, assume SABCD = SABCD, thenb) IfT，< T+x,Snaco-Saco = \(tg).x-f[ λ(t)dt=0pR(x|T)>R,(x1T)ifM<0R(x1 T) = R,(x1 T) ifM =0(4)lr.(x1 T) 0Accordingto Eq. (4)， 3ig, so thatMost of the cases of software reliability are similarλ(tg) =一厂\(1)d.to those of Theorems 1 -4, and lttle software is re-leased when the failure ratio is still increasing, there-Assume the reclangular area surrounded by λ。andfore, the second case of Theorem 4 seldom occurs, andthe time[s,s +x) is S。, thenRop(x1 s) = exp[-S。](5)it is included for the sake of completeness.Therefore, the dscusn is made acording to the 3 Infuence of the Testing and Operational Reliafollowing three cases:bilities on the Software Optimal Release(1)wheni < tg, λ。= λu,S。> SABco(2)whent = tg, λ。= λ(lg),S。= SARcDAfter the software is developed, the quality re-(3)whent > lg, λ。=入u,S。< SAacD(6)quirement is the key deciding whether the software canbe put into practice. An important characteristic is the2 Comparison between the Operational Reliabilitysoftware reliability, and the longer the software testingand Testing Reliabilitytime is, the higher the software reliability is. Mean-while, the software testing has a great influence on theFrom the viewpoints of users, the operational reli-developing cost and the distribution time of software. Itability is what users concerm. However, most of theis important when to stop testing the software to be re-software reliability growth models give the testing relia-leased and whether the released software is reliable inbility. In the testing phase, the mean function m(t) isthe operational phase. The operational reliability isexponential, that is, the failure ratioλ(t) is monotoni-what users concerm, thus, using different reliabilitycally decreasing; or, the mean function m(t) is S-concepts will affect the optimal release and the testingshaped, that is, the failure ratio入(1) is firstly increas-cost of the software.ing and then decreasing. The fllowing theorems areThe total cost of the software during the testinggiven in terms of two forms of the failure ratio.and operational phases is denoted by C(T), then theTheorem1 For anyT≥0 andx > 0, whent expressed as:SAacD according to Eq. (6), Rop(x1s) =exp[-S.] 0, whent =中国煤化工，(9)tg,ie. , λo'= λ(tg), ifλ(t) is strictly decreasing forntion constitute theanyt, Rp(xI T) = R,(x1 T).optim:MCNMHGsting phase, andTheorem3 For anyT≥Oandx > 0, whent >Eqs. (7) and (9) in combination constitute the opti-tg,ie., λ。= λad，ifλ(t) is strictly decreasing fomal release prineiple of the operational phase. Now,●347●Journal of Harbin Institute of Technology (New Series), Vol. 15, No. 3, 2008the following definitions are given:C(T), Ti = T, and the testing phase needs to satisflyTh: Satisfying the minimum value of Eq. (8),Tpi = max{T.,Tk}.T≥0.3) IfRg(x10) < R,, since the optimal solutionsTR: Satisfying the minimum value of Eq. (9),of the testing phase and the operational phase are limit-TR≥0ed by the reliability level, Tp: = max{T,Tk},Tpz =Tc: Satisfying the minimum value of Eq. (7)，max{T,Ti}.Tc≥0.The dermonstrations of Theorems 5 and 6 are simi-Tp: Satisfying the opimal releasing value of tes-lar to that of Theorem 7.Theorem8ForanyT≥0andx>0,whenting phase.Ti: Satisfying the optimal releasing value of oper-λ(1) is firstly increasing and then decreasing, assumethat T。represents the inflection point of the mean func-ational phase.Theorem5 WhenI < tg, i.e. λ。= Aw, iftionm(t). IfT。≤T, whent > lg, ie. λ。= λod，λ(t) is strictly decreasing for any t, there are three ca-1)IfR_(xI T)≥Ro,Tpi = Tμ = max{T,To}ses as follows:2)IfR.(x1 T,) ≥R > R(x1 T), Tp =1) IfR.(x10)≥Rq，Tj=Tμ= T。max{T,Tk},Ti = max{T,To}2)IfR_(x10)≥Ro>R(xl0),Tj = T,Ta)HfT.≥Tl,Tj=Ti= T= max{T,TR}b)IfT。< T。< Th, T=T. 0, whenTheorem6 Whent = lg,i.e. λ。= λ(rg), ifλ(t) is firstly increasing and then decreasing, assumeλ(t) is strictly decreasing for any I, the cases are samethat T。represents the inlection point of the mean func-as those of Theorem 5.tionm(t). IfT。≤T, whenl≤lg,i.e. Rgp(x| T,)≤Theorem7 whent > tg,i.e. λ。= λd, ifλ(t)RQ(xI T,),is strictly decreasing for any t, there are three cases as1) IfR_(x| T,)≥R, Tpi =Ti = max{T,T}follows:2) IfR(x1 T,) ≥R > R_(x1 T,),Tji =) IfR.(x10)≥Ro,Tp° =Tj = T.max{T,To},Ti = max{T,Th}2)IfR(xI0)≥R。>R。(xI0)，Tpi=a) IfT.≥T，Tj= T2 = Tcmax{T,TiL,Ti = r。b)IT。 tg, i. e.lar to that of Theorem 7, and thus are omitted.It can been seen from Theorems 5 and 9 that,λ。=λad, according to Theorem3, T≥0, andx > 0,R.(x1 T) > R(x1 T)1) whent ≤tp and \(t) is monotonically decrea-1)IfR.(x10)≥R,R(x1 T) >R_(x1 T)≥sing, i.e. Rp(x1 0)≤Ru(x| 0), orλ(t) is firstlyRo. Since the optimal solutions of the testing phase andincreasing and then decreasing, i.e. Rp(xI T)≤the operational phase are not limited by the reliabilityR(x| T), Tpi≤Tμlevel,Tp: = Tj = T。2) whent >t= and λ(t) is monotonically decrea-2) IfR.(x1 0)≥R。> R_(xI 0), similar to .sing,中国煤化工, or \(t) is frstlycase 1), the operational phase satisfies the reliabilityincreMYHCNMHGR.(x1 T。) >requirements, and thus the optimal solution of the op-Ru(x 0σ，42- 'p.erational phase only needs to satisfy the condition ofGenerally, according to Eq. (1), 入。≤hr,●348●Joumnal o[ Harbin Institute of Technology (New Series)， Vol. 15, No. 3, 2008R._[-λ。x]≥R.[- λrx], that is, the actual relia-ifλ。=入s=λ(i=70)=0.028,itcanbeobtainedbility level of the operational phase is greater than thefrom the item3 of Theorem7 that, ifT.≤TR,Tpz =reliability level estimated in the testing phase. The re-r员=57.86lg,ie. λ。=λu，In the actual software testing, if the diference be-，349●Journal of Harbin Instiute of Technology (New Series), Vol. 15, No. 3, 2008tween the testing environment and the operational envi-IEEE, 2003. 66-70.ronment is taken into account, the failure ratios of the2] Zhang Xuemei, Teng Xiaolin, Pham Hoang. Consideringfauht removal efficiency in software reliability 8s8essment.software will be different, and thus the testing reliabili-IEEE Transctions on Systems , Man, and Cybernetics , 2003,ty and the operational reliability will be different,33(1):114-119.which will affect the optimal release time and the cost[3] MusaJ D. Operational profiles in software reliabilit engi-of software. The above discussion only ilustrates thatneering. IEEE Software, 1993 ,10(2): 14 -32.different kinds of reliabilities will have an influence on4] MusaJ D. Senitivity of field failure intensity to operationalthe optimal releasing time and the cost of the softwareprofile erors.Proceedings of the 5th Intemational Symposi-um on Software Reliability Engineering. New York: IEEE,according to the cases possibly occuring during the ac-1994.334 -337.tual software testing, but not means that these cases are[5] Kenney C Q. Estimating defects in commercial softwareonly applicable to the present example and the Goel-during operational use. IEE Trans on Reliability , 1993 , 42Okumoto model. The mathematical discussion of the(1):107 -115.testing reliability level and the operational relability[6] Chen Y. Modeling software operational reliability via inputlevel and the analysis conclusions in this paper willdomain-based reliability growth model. Proceedings of thehelp the software testers to select reliability criterions28th Intermational Symposium on Fault-Tolerant Computingaccurately and furher decide the optimal release time(FTCS). New York: IEEE, 1998. 314 - 323.of the software, thereby optimizing the resource alloca-[7] Musa J D. Adjusting measured field failure intensity for operational profile variation. Proceedings of the 5th Intema-tion.tional Symposium on Sftware Reliability Engineering. NewYork :IEEE, 1994. 330-333.5 Conclusion[8] Li Naixin, Yashwant K. Malaiya. On input profle selectionfor software tesing. Proceedings of the Intemational Sympo-According to the similarities and differences be-sium on Software Reliability Engineering ( ISSRE). Newtween the testing phase and the operational phase , sev-:IEEE, 1994. 196 - 205.eral different formns of the comparison between the fail-[9] Zhang Xuemei, Jeske Dr, Pham Hoang. Calibrating soft-ware reliability models when the test environment does noture ratio of the operational phase and that of the testingmatch the user environement. Applied Stochastic Models inphase for non-maintenance system are analyzed in thisBusiness and Industry, 2002 (18) :87 -99.paper, and mathematical discussions and analysis are[ 10]Yang Bo, Xie Min. A study of operational and testing reli-performed in detail. The operational reliability is whatability in software reliability analysis. Reliability Engineer-users concem , using different reliabilities has an influ-ing and System Safety ,2000 (70) :323 -329.ence on the optimal release time and the cost of the[11] ANSLIEEE. Standard Clossary of Software Engineeringsofware. Finally, a verification is performed on a cer-Teminology. STD - 729 - 1991, ANSI/IEEE.1991.tain data set using the classic Goel-Okumoto model by[ 12]Lu Minyan. Study on software reliability parameters. Jour-nal of Beijjng Univesity of Aeronautics and Astronautics,an example, and two kinds of conclusions about the in-2001 ,27(2) :242 -244.fluence of using different reliabilities on the optimal re-[13]Goel A L. A sofware error detection model with applica-lease time and the cost of the software are derived, oneions. 2d Software Life Cycle Management Workshop. At-is to continue testing to meet the reliability level re-lanta: 1978.quirements of users, and another is that testing stops[14]Misra P N. Software reliability analysis. IBM Systems Jour-when the operational reliability level requirement isnal, 1983 ,22(3) :262 -270.met, thus the testing cost can be reduced.[15]Kazu Okumolo, Goel A L Optimal release time for soft-ware systems. Proe IEEE Computer Society IntemationalReferences:Computer Software Applcation Conference 3*. New York:[1] Yang Shiping, Nan Sang, Xiong Guangze. Integrated safety[16]Xie Min, Yang Bo. A study of the efet of impereet decritical systems on reliable real time network. Proceedingsbugging on software development cost. IEEE Transactionsof IEEE Intemational Conference on Parallel and Distribu-on Software Engineering, 2003 ,29(5) :471 - 473.ted Computing, Application and Technologies. New York:中国煤化工MYHCNMHG●350.