Sunday, June 15, 2014

Be a Quality not a Tester!

Generally speaking I don't like someone to call me as a tester! , because I'm a quality and the difference between the two words are big –I think the same meaning is the difference between being coder and being developer!-

Being a quality is required additional analysis skills for what you discover, not only acting as a bug finder, but analysis these defects and take a proper action according to your analysis results.

Of course using a tool will facilitate this to you, but you can survive also if you don't have one.

In my xCompany we don't have such reporting tool so I create this using Excel! , just put your defects in the master sheet and the powerful excel will generate all the graphs to you.

Use this tool as a weapon in you status meeting against any enemy.. Sorry I mean against any project manager or a developer J

I'd like to introduce this effective tool [based on MS Excel] that will help teams and project managers to control the quality through analyze defects.

In brief:

- The tool is auto generated automatically [attached].

- The tool is very effective in stand up meetings to discuss the big picture of defect status.

- The tool make some analysis to defects like :

o Defect prediction model(Which very vital report to expect that we will met the delivery date or we
need more effort to close the open defects)

o Defect distribution

o Defect submitter per severity

o Defect per state

o Defect classification

o Weekly progress

Finally , using this tool will shift the QC team from being testers only to being a real quality team and I recommend to institutionalize it across your organization.

Finally Email me to get the tool! , Ahmed.Ibrahim@yahoo.com

Sunday, July 1, 2012

Security Testing Using Acunetix

Security Testing !

Security Testing now a days is not a luxury ! it's essential for each project release to pass through this phase , below is one of the most powerful scanning tool in the market , it's a amazing tool with less headache in runnung and reporting.

http://www.acunetix.com/vulnerability-scanner/

Acunetix is :

An automatic client script analyzer allowing for security testing of Ajax and Web 2.0 applications
Industries' most advanced and in-depth SQL injection and Cross site scripting testing
Advanced penetration testing tools, such as the HTTP Editor and the HTTP Fuzzer
Visual macro recorder makes testing web forms and password protected areas easy
Support for pages with CAPTHCA, single sign-on and Two Factor authentication mechanisms
Extensive reporting facilities including VISA PCI compliance reports
Multi-threaded and lightning fast scanner crawls hundreds of thousands of pages with ease
Intelligent crawler detects web server type and application language
Acunetix crawls and analyzes websites including flash content, SOAP and AJAX
Port scans a web server and runs security checks against network services running on the server

more over , it's freeeeeeeeeeeee !!

Thursday, October 14, 2010

Test Automation Process & Guidelines

Objective

The main objective of this article is to set some guidelines to guide the automation process in any project.

This document will present the automation process sequentially from the beginning (ROI for the project, schedule and estimation, execution, and maintenance)

for any further help or to get the template please contact me Ahmed.Ibrahim@Yahoo.com

Should we automate the testing?

First of all we must answer this question , we will doing that by conducting ROI on the project then let the upper management to decide should we automate it or not?

Prepare for ROI

There are some questions must be answered before doing the ROI:

What type of application are you automating? (Java , .Net , HTML)

Number of test cases executed per release?

Estimated cost to automate each test case?

Number of testers per release needed for manual execution?

Number of releases per year?

Days of testing per release?

Average tester hourly rate?

Number of Rational Functional Tester Licenses needed?

2.Conduct ROI using IBM ROI Calculator
http://www-01.ibm.com/software/rational/offerings/testing/roi/tool/ROI_Rational.html

3.Agree on ROI Results

The upper management must agree on ROI analysis result in order to resume the automation project, and must be updated regularly with the measurement and results from the automation.

Develop Test Automation Plan

1. Target test types to be automated (Regression, Smoke, System test)?
2. What is in scope and out of scope modules?
3. List of all selected "Test cases" to be automated.
4. Estimation
  
  The following are factors impact on the estimation for the project:

#	Type	Factor	Impact	Remarks
1	Framework	Availability	High	Good framework makes your scripting, debugging and maintenance easier. Do understand that framework needs continuous updating across the script development.
2	Application	Repeat functionality	High	It is quite easier to automate incase the functionality repeats across the application (Recommend to use keyword driven in such cases, as you do not end up in writing more action/verification based methods). If NOT, then the effort of building libraries and/or scripts is more linear in nature.
3	Project	Test Scope	High	If the complexity of application as well as the test scope is complex in nature, then it would consume huge efforts to automate each test case.
4	Test tool	Support to AUT	Medium	The test tool to be used may not support some application functionality and may cause overhead. You may find it more difficult to get started with open source scripting and/or tools
5	Application	Custom Objects	Medium	The number of custom objects in the automation scope matters as it becomes overhead for the test automation team to built and maintain the libraries for them.
6	Scripter	Skills	Medium	This costs project. The right skill packages of the scripter are very essential for any good test automation. If the customer NOT willing to provide the leverage on the estimate for this factor, do NOT forget to add the learning curve cost to the overall time.

While estimate the TCs : the Estimator must be sure that he take into his consideration the following aspects for each test case

SUB COMPONENT	Estimated Efforts			Remarks
SUB COMPONENT	Simple	Medium	Complex	Remarks
Pre–Script Development
Test Case execution (Manual)	0.5	1	2	For 1 iteration (assuming scripter knows navigation)
Test data selection	1	2	3	For one data set (valid/invalid/erratic kind)
Script Template creation				Can use script template generation utility to avoid this
Identify the required reusable (Design the TC before start)				Assuming proper reusable traceability matrix presence
Script Development
Application map creation				Assuming the no of objects = number of actions
Base scripting				Normally all these go hand-in-hand. Separated for analysis & reasoning.
Add error/exception handling
Implement framework elements
Script Execution
Script execution				For n iterations (~ average iteration count)
Verification & Reporting				Assuming there will minimal defect reporting.
Total Effort per script	15	30	50

Test Schedule and Resources planning

Establish VSS environment to put test automation scripts under it

Test Design
1. Design every test case by defining the global functions to be used
2. Following Design Template

Coding Standards

Naming conventions
1. Script naming conventions
2. Result naming conventions
3. Report naming conventions
4. Execution log design

Script, Result and Report Repository
1. Location for the repository
2. Repository access right

Test Automation environment details
1. Software require
2. Hardware require

Test team roles and responsibilities

Role	Responsibilities
Automation Project Sponsor	Approve project ROI, handle major issues related to automation project development, and approve on required resources
Automation Project Manager	Manage the overall development and execution of the automation project Obtaining resources for the project Identify risks and handling major issues Approving technical design and overall timeline Delivering the overall scripts Preparing for training if needed Prepare test automation environment Provide project schedule and project plan Track automation project development status Manage changes and issues
Test Automation Tem Leader	Provide Technical guidance to the Test Automation Team Conduct code review and design review on the conducted work. Select Framework and ensure that all the team follow that framework Handle day to day tasks
Test Automation Designer
Test Automation Developer	Develop scripts based on test case design Learn and study the application under test. Maintain Traceability Matrix , Global Functions Matrix Follow test automation framework Fix any defects found during the execution.
Test Automation Officer	Responsible for running test scripts Log defects on tracking tool (the defect )Tracks SCR's in DEFECT TRACKING TOOL. Reviews new SCR's for duplicates, completeness and assigns to Module Tech Leads for fix. Produces status documents as needed. Tests changes, logs incidents identified during testing.

Test team training requirements

Automation Process

Design Phase:

1-Roles and Responsibilities

Designer

Grouping similar test cases.

Preparing question lists for every test case to discuss with the analyst.

Conducting Knowledge Transfer Meeting (KTM) with the analyst for certain set of groups according to analyst specialization.

Refine the test case design with the result of the KTM.

Complete the design document for each group of test case
1. Define Existing function for every Group of test cases
2. Define Data pools for every function
3. Define Script Type For Every function
4. Define Data and Data Type input for every function
5. Define the verification point for every test case
6. Conduct a peer review on other designers
7. Conduct a technical review for the coding against the design

Analyst:

Attend Knowledge Transfer Meeting with the designer.
Provide more detail for any ambiguous steps or calculations in the test case.

2-Entrance Criteria

For Existing test cases:

The entrance criterion for the design phase is a valid and reviewed test case and categorized by group according to the business

For Updated Test cases:

The entrance criterion for the design phase is any updates done on the base lined test cases.

3-Input

- The test cases which are reviewed by the analyst and Base lined

- TCTM

4-Tasks

4.1 Requests for Checkout Test Cases

Inform the configuration controller to check out desired test cases by mail

4.2 Create or Update Design Document

To create the design document from scratch or update it if exist Please see the design template

.\DesignTemplate.doc

4.3 Update Tractability Matrix

The traceability matrixes which bind every test case with design document with developed script please see the Traceability Matrix:

.\Automation_TM.doc

4.4 help the developer to achieve high level of test case understanding

After studying the test cases and conducting the knowledge transfer meeting with the analyst, the designer must be able to clarify any ambiguity in the test case to the developer to help him to achieve a quality code

4.5 Conduct Reviews

4.5.1 Peer Review

Conduct a peer review for other designer on the design document as per quality plan.

4.5.2 Business Review

For every developed script, the designer must re execute the script many times to ensure the start status and the end status of the script

4.5.3 Technical Review

Review the code against the Automation Code Standard Document.

Review all types of script that the developers employ it in the code.

Review all Data pools contents or any other data retrieval method in the code.

4.5.4 Log Review

Review that the script log to be clear for the executer

5-Output

5.1 Questionnaire Document

Prepared by the designer to discuss it with the analyst in the KTM –embedded in the design document -

5.2 Design Document

For every set of test cases (Test Case Group)

5.3 Reviews Documents

For the above mentioned reviews

5.4 Updated Traceability Matrix

By the end of reviewing the design document x

6-Exit Criteria

For each test case, the design end is reached when the designed test case has been already tested, reviewed and base lined.

Development Phase:

1-Roles and Responsibilities

Developer:

Build the script for every test case upon the design document.

Create required data pool as the design document.

Conduct a unit test for the developed script.

Designer:

Technical consultation for every group of test cases.

Review the developed script as mentioned in the design phase section.

2-Entrance Criteria

Based line the design document will trigger this process to start.

3-Input

Traceability Matrix: to be updated by developed script.

Design Document: to be followed by the developer.

Maintenance Matrixes:

Business Function /Sub routine Matrix

The purpose of this function to illustrate which subroutine used in which business function

Test Case\ items Matrix

The purpose of this function to define which test case will be affected by changing certain item

4-Tasks

4.1 Develop the test case script

According to the design document the developer must develop the required scripts and required data pools according to the coding standard

Please see the coding standard at the path:

Put the link here.

4.2 Conduct a unit test for the developed scripts

4.3 Update Maintenance Matrixes

4.3.1 Traceability Matrix

4.3.2 Business Function /Sub routine Matrix

4.3.3 Test Case\ items Matrix

4.4 Conduct a unit test for the developed scripts

5-Output

Developed Script

Updated Traceability Matrix

Updated Maintenance Matrixes

6-Exit Criteria

Check in the scripts in the source safe tool.

Sunday, September 19, 2010

Extract All Cases from Requirement using Cause and Effect Analysis Technique

Extract All Cases from Requirement using Cause and Effect Analysis Technique

Introduction:

This article describe in details an organized method to extract all possible cases from the requirement , this can help the system analyst in defining all the cases and also can help developers while preparing the unit TCs and of course will help the testers to write unified test cases.

First of all suppose you have the following requirement:

A mailing is to be sent out to customers.

The content of the mailing is about the current level of discounting and potential levels of discounting. The content is different for different types of customers.

Customer Types A, B and C get a normal letter except Customer Type C, who get a special letter. Any customer with 2 or more current lines or with a credit rating of ‘X’ gets a special paragraph added with an offer to subscribe to another level of discounting.

The problem is if you give this requirement to any 4 different testers, the 4 will produce different no of TCs, I will explain how to derive a unify number of TCs using cause and effect analysis and decision table.

Step 1: Break the specification down to cause and effect:

First the analyst must break down the requirement to causes and effects

– Cause: is a distinct input condition or an equivalence class of input conditions (the cause which trigger the effect) in our above example Customer Type (A,B,C, Other) is a cause

– Effect: is an output condition. Determine the logical relationship between the cause and the effects , in our above example “Get a normal letter” is an effect for the cause customer type(A,B,C, Other )
Usually the effects is an output action (Ex. Error message , application action , executing specific application job , etc…)

After this step you should have cause and effects like that:

C1: Customer Type A,B,C, Other

C2: 2 or more lines

C3: Credit Rating

E1: Normal Letter

E2: Special Letter

E4: Add special paragraph

E5: Get Normal letter

Step 2: Put Causes and Effects in the decision Table.

Then we should fill the decision table by putting the cause and the effects together in that table as a following:

Cause\Effect
C1:
C2:
C3:

E1:
E2:
E3:
E4:
E5:

Step 3: Calculate all possible Combinations

This is very important step to know no of columns in our decision table, simply use this formula to know all no of combinations that can be occur:

• If all causes are simply Y/N values:
2^{number of causes}

• If 1 cause with 3 values and 3 with 2:
3¹ * 2³ = 24

In our example, C1 has 4 values (A or B or C or Other) AND C2 has 2 values (YES or NO) also C3 has 2 values (YES or NO) so our formula should be :

1⁴ * 2² = 16 combinations

So we should draw for the decision table 16 columns which present 16 different combinations.

Now the table should be like that:

Cause\Effect

C1:

C2:

C3:

E1:

E2:

E3:

E4:

E5:

Step 4: Fill Columns

We can fill columns 2 methods:

1- Logical method: in this method you start filling column by column , start with the causes and end up with the effect (put into consideration that every column is a case).

2- Repeating Factor Algorithm method:

a. Determine Repeating Factor (RF): divide remaining combinations by the number of possible values for that cause

b. Write RF times the first value, then RF times the next etc. until row is full

Example for using Repeating Factor Algorithm: suppose you have the following decision table:

	1	2	3	4	5	6	7	8
C1(Y\N)
C2(Y\N)
C3(Y\N)

E1
E2

Fill first row: RF = number of remaining combination (8) / possible value for this

Cause (2) = (8/2 = 4) , so we will fill 4 from Y and 4 of N , the DT will be :

	1	2	3	4	5	6	7	8
C1(Y\N)	Y	Y	Y	Y	N	N	N	N
C2(Y\N)
C3(Y\N)

E1
E2

Fill Second row: RF = number of remaining combination (4) / possible value for this

Cause (2) = (4/2 = 2) , so we will fill 2 from Y and 2 of N , the DT will be :

	1	2	3	4	5	6	7	8
C1(Y\N)	Y	Y	Y	Y	N	N	N	N
C2(Y\N)	Y	Y	N	N	Y	Y	N	N
C3(Y\N)

E1
E2

Fill Third row: RF = number of remaining combination (2) / possible value for this

Cause (2) = (2/2 = 1) , so we will fill 1 from Y and 1 of N , the DT will be :

	1	2	3	4	5	6	7	8
C1(Y\N)	Y	Y	Y	Y	N	N	N	N
C2(Y\N)	Y	Y	N	N	Y	Y	N	N
C3(Y\N)	Y	N	Y	N	Y	N	Y	N

E1
E2

Filing DT in example1:

In example 1 we have 3 causes and 5 effects with total number of combinations = 16 combinations

Fill first row: RF = number of remaining combination (16) / possible value for this

Cause (4) = (16/4 = 4) , so we will fill 4 from every type (A,B,C,O) , the DT will be :

Cause\Effects

C1(A,B,C,O)

C2(Yes\No)

C3(yes\No)

Fill Second row: RF = number of remaining combination (2*2) / possible value for this

Cause (2) = (4/2 = 2) , so we will fill 2 from every Y and 2 from N , the DT will be :

Cause\Effects

C1(A,B,C,O)

C2(Yes\No)

C3(yes\No)

Fill Third row: RF = number of remaining combination (2) / possible value for this

Cause (2) = (2/2 = 1) , so we will fill 1 from every Y and 1 from N , the DT will be :

Cause\Effects

C1(A,B,C,O)

C2(Yes\No)

C3(yes\No)

Step 5: Reduce Combinations

1- In our simple example find a don’t care “-” sign and place it ,

	1	2	3	4	5	6	7	8
C1(Y\N)	Y	Y	Y	Y	N	N	N	N
C2(Y\N)	Y	Y	N	N	Y	Y	N	N
C3(Y\N)	Y	N	Y	N	Y	N	Y	N

E1
E2

In the above DT suppose in column 3 and column 4 , when the C2 become “N” the program end , so we will put in C3 the sign “-” which mean done care , the DT will become as follow :

	1	2	3	4	5	6	7	8
C1(Y\N)	Y	Y	Y	Y	N	N	N	N
C2(Y\N)	Y	Y	N	N	Y	Y	N	N
C3(Y\N)	Y	N	-	-	Y	N	-	-

E1
E2

2- After placing the “-” don’t care sign , now remove the repeated columns , column 3 and 4 now become identical so will remove one of them , also column 7 and 8 now identical so I will remove one of them too , so the DT will be :

	1	2	3	5	6	8
C1(Y\N)	Y	Y	Y	N	N	N
C2(Y\N)	Y	Y	N	Y	Y	N
C3(Y\N)	Y	N	-	Y	N	-

E1
E2

After removing the columns you must do a checksum for the table to assure that the DT still

cover all the original combination = 8

To do checksum:

• For each column calculate the combinations it represents

• A ‘-’ represents as many combinations as the cause has

• Multiply for each ‘-’ down the column

• Add up total and compare with step 2

So the check sum will be as follow:

	1	2	3	5	6	8
C1(Y\N)	Y	Y	Y	N	N	N
C2(Y\N)	Y	Y	N	Y	Y	N
C3(Y\N)	Y	N	-	Y	N	-

E1
E2
Checksum	1	1	2	1	1	2	8

Step 6: Add effects to table

• Read column by column and determine the effects

• One effect can occur in multiple test combinations

Cause\Effects

C1(A,B,C,O)

C2(Yes\No)

C3(yes\No)

E1: Normal letter

E2: Special Letter

E3: Add paragraph

E4: No Letter

Checksum

As we see in the above table after adding effects to the table you can:

1- Determine if you have missing requirement or not as the case we place “?” , the requirement is missing if the customer type is other than the 3 types A,B,C

2- You can determine if there is a redundant requirement , example in the column 6 and 7 by looking into the table you will find the 2 columns has the same effects , so you can ask the client or the analyst about this situation and if it doesn’t matter to have credit line or not in case of customer type B

3- Finally you will be sure that you cover all possible scenarios by doing this analysis.