2007, Q6 - Question on Testing

[Hort]

Hi,

I have had a go on a question involving testing. There is not a specific thread set up for this type of question so I posted it here. I don't have "hands on" experience of testing, my knowledge is based on reading standards and reviewing test plans & strategies so would be interested to see how close/far I'm off the mark!

H

[PJW]

Hi,

I have had a go on a question involving testing. There is not a specific thread set up for this type of question so I posted it here. I don't have "hands on" experience of testing, my knowledge is based on reading standards and reviewing test plans & strategies so would be interested to see how close/far I'm off the mark!

H

You are right; I agree that there should be a specific category set up and hence I have now done so and moved.

============================================================================================================

My initial view of the answer was that it did not answer the question; however when I got to the end I could see that there was then some attempt to do so. When I read it again, having also re-read the full question, I did warm to it appreciably.

However I feel that you made a mistake in how you presented your answer. As in the commented version attached, a short intro describing your approach for answeering might have made things clearer to the examiner. I do now see what you were trying to do and avoid duplication where there is similarity but, even so, having so little text relating to the differences is a bit of a high risk strategy and the necessary placement at the end so that the question seems to be answered backwards to some extent probably wasn't a great idea.

I think you'd have given a very much better impression by a small modification of your approach. Had you provided 3 narrow columns for i), ii) and iii) alongside your broadly chronological description of the testing process in which you could have put Y = required, N= not applicable then it would have seemed from the start that you were really addressing the question.
Then when you wanted to give more detail at the end re perhaps how much could be done as preparation and how much during the main commissioning you could have had some numbered items and indeed put cross references to those numbers back in those columns alongside the Y or N to link the comment to the specific element to which the comment relates.

As far as the content itself was concerned then what was written was basically accurate and reasonably described (I am of course assuming a Netowrk Rail context here); I could certainly correct / refine quite a lot but then this area is one in which I have spent much of my last 30 years. Judged on the basis of an overview for IRSE Exam purposes then it is not bad and it certainly isn't obvious that this is largely book-learning without much real experience so you have done well. Yes there were some ommissions and imperfections but the only thing that really jarred as being misunderstood is the meaning of "controlled changeover" which I have attempted to clarify a bit.

I suppose that the most significant comments are about the scope; you have effectively covered New Works site testing reasonably well and have pointed out some of the significant differences relating to item iii) for which the Signal Maintenance Testing is applicable.
However:
a) you didn't make enough of the fact that SMT is based on the premise that the system as a whole used to work and that just one or a small number of components is being renewed effectively like-for-like to restore the system as a whole into its pre-existing state (perhaps a polanned renewal but more often the rectification of aa failure). For that a lot can be taken for granted, the task is primarily to ensure that the site drawings initially seem to reflect reality and then the part is exchanged and then its interconnections reinstated as they were before and then the item given a functional test to prove its basic operation. There are generic "Test Plans" for each likely scenario relating to each item of equipment and the relevant one(s) are selected and implemented.
Often a small team of typically 3 people would identify fault, undertake unit exchange and then independently test, They use the existing maintenance drawings for reference but not record on them; the recording is in a Log Book noting the steps completed for each of the generic Test Plans.
The contrast with the New Works approach is that there are specific Test Plans written for each possession and in general there are multiple teams performing a whole range of testing on a system that has never previously been operating precisely how it will be at the end of the testing activity. There are Test Specifications and Audit Sheets, but the actual testing tends to be more variable and less prescriptively defined re what is to be undertaken in what sequence, although the Testing Plan for the shift / period may give further detail and indeed the Tester in Charge on the shift will be taking the overall decision re who is to do what when to minimise conflicts between all the different work that could mutually conflict- in reality much is often left to the experience of the various testers to come to a mutual understanding as things develop during the performance of the works.

b) The other very significant issue, which is also far more applicable to i) and ii) than iii) are all those activities which form part of V&V but are not within the direct scope of site testers. You made the mistake of interpreting the question as being purely about testing whereas Verification and Validation is far wider than this; it encompasses the whole traceability that assures that the finished complete system correctly does what it was specified to do. Even in very traditional implementations, there is a lot that preceedes the "testing", including:
1) the quality type issues that are the province of the installation supervisor, the traceability of each length of cable installed to the drum and therefore the manufacturing batch, the Quality Assurance process re the manufacture of the simpler items and the Certificates of Conformity for the more complex assemblies delivered from the factory,
2) the design process: production check, independent check, CRE Approval, Inter Disciplinary Reviewofrelevant design where there are interfaces, Client Acceptance, "Approved For Contstruction"
3) product approval be it "grandfather rights", inclusion on PADS database having had some form of safety acceptance, use withn application constraints etc
The question didn't specifically ask re novelty, but the reality is that almost every project nowadays uses some new variant of something or uses things in a new combination, environment etc. so some reference to this is certainly pertinent.

I would have concentrated much more of the answer in describing all these sorts of things where the categoories i - iii were different and would therefore have had to have been briefer re the similarities- this is because I feel that this was what the question was really seeking to explore.

I must admit that I do have a bit of a problem with item i) in the question; there is always something that needs changing! In all my career I have never worked on totally new "green field" railway, although in some ways putting MAS TCB on a line controlled by mechnaical signalling can be quite like this in that the new system is very largely independent of the old and can all be got working early, with just the signals being uncovered on the actual day (assuming that the new points can be laid in prior and also pre-tested but then clipped out of use). Similarly implementing ETCS L2 with axle counters on a line previously operated under RETB also gets close. Where there is a completely new line such as CTRL / HSL1, then this is truly what is really meant by i), but even so there will be interfaces with existing railway at then extremities and hus several "fringes" and in those areas at least the scenario is as ii), although I accept that it is a small proportion of the whole length. I suggest that it would only be a Metro line that might just be completely i), but even so these are rarely completely isolated and somewhere along the line there will be a connection to some other railway on which the signalling is not being changed- definitely the case with the LU network that also has several connections / joint running lines with Network Rail, also think of the various trams in Croydon / Manchester / Newcastle etc. So I feel that there is no great distinction between i) and ii), more a matter of degree that things might be done pre-test rather than during commissioning period. I guess also that the more nearly that the circumstances approximate to i), then the greater the degree of novelty may be being implemented on the project (the freedom from interface certainly makes this a more praticable option than it would otherwise be) and therefore the greater emphasis that would generally be applied to early proof of concept by modelling, early trilals etc to gain some interim safety acceptance prior to the main testing phase.

Summary
Overall then I believe that you didn't appreciate the full scope of the question, but had only interpreted quite narrowly.

Without any change to the material, you could have made a better job at aligning to the wording of the question.

Certainly I think there should of been more explanation of why there are so many different elements of testing, what they all contribute (relate to what risk they are a mitigation and how they achieve this).
Having said that then it was at least competent and I eventually concluded reasonably aligned, so it'd be a Pass I think.

[Peter]

Hi,

I have had a go on a question involving testing. There is not a specific thread set up for this type of question so I posted it here. I don't have "hands on" experience of testing, my knowledge is based on reading standards and reviewing test plans & strategies so would be interested to see how close/far I'm off the mark!

H

I had very similar thoughts about whether you had answered the question that was asked but on re-reading the question changed slightly too. PJW has said much of it, but just to confirm, I think the answer was presented in the wrong order. This was a pity as it was good stuff, perhaps just not the right good stuff.

I thought that the bit on maintenance testing should have more clearly emphasised the fact that you are only testing around the bit that has been affected. This could have been brought out in an explanation of when maintenance testing can be done (it could be part of a new scheme where, say, a signal head is being updated but kept in the same place) and that there is the need to test "just before and just after" the alteration. As PJW said, also, the "previously in and working" premise is very important.

Another area in the works testing bit that I thought would have been in scope of the question would to be consider some of the strategies like "over and back" testing. You tended to assume that what was new could never be tested until the day of the race in the middle scenario.

[Hort]

P&P,

thanks for the pointers. You are right in that I took that approach to answering the question to avoid duplication, however, i need to be aware of how the answer ris read by the marker. So to be clear, in order to fully answer the question concisely within the allotted time, would you suggest scaling back the content relating to testing and concentrate more on design and installation quality assurance and product approval? There is a lot to write about this topic so keeping it concise would be a challenge.

At least I'll take comfort in my overview of the testing process being largely accurate.

[PJW]

P&P,

So to be clear, in order to fully answer the question concisely within the allotted time, would you suggest scaling back the content relating to testing and concentrate more on design and installation quality assurance and product approval? There is a lot to write about this topic so keeping it concise would be a challenge.

You need to demonstrate that "Verification and Validation" is much wider than just testing.
On the basis that I believe that in cases of doubt that BREADTH is more important than DEPTH, then yes scale back detail of content re testing.
In the maintenance context then really it very largely ONLY site testing for verification; this in itself is perhaps the most significant difference and hence one of the main things that the question was seeking.

I would expect that a good answer would at least touch upon;
a) Client business needs as expressed via Project Specification / User Requiements
b) Derivation of the Functional / Non-Functional / Interface Requirements that define the project
c) Traceability of these, the assignment / apportioning to the various sub-systems, products
d) Detailed independent checking of design
e) Overview review of design concept, compatibility check of interfaces
f) Analysis, Modelling, Simulation etc
g) Enviromental and other qualification tests for items of novel equipment, pilot trials etc
Quality control of materials, Quality Assurance of products,
h) Certificates of Conformity of Equipment
j) Individual subsystems off-site verification test of software and hardware
k) Software and hardware integration for each sub-system separately
l) Integration of various sub-systems so that they operate together as a larger system
m) Verification that the complete system meets the design requirements both functionally and for RAMS etc. (which may well require a period of time after being taken into operational service to fully demonstrate)
n) Validation that the the implemented system fulfills the actual need which initiated its specification in the first place
p) Experience of operational use: support- spares, training, reliability growth etc

However having painted the broad brush picture of "the full Monty" and indicated how much of this applied in the scemnarios listed, would definitely concentrate on illustrating what this theoretical description would mean in reality and particularly concentrate on the activities that are most relevant to a traditional signalling system and therefore give examples that relate primarily to the site testing phase (as indeed you did in your answer).