Skip to main content

How to Optimize Data-driven Automation with DesignWise

What you'll learn
- Benefits of data-driven automation with DesignWise
- Creation of compatible scripts in Playwright and Cypress

 

Overview

Why data-driven automation can be a good choice

Definition
Data-driven testing is the practice of keeping the execution script separate from the test data. In other words, you will have a single automation script that will reference a data table containing multiple rows with scenarios.

When we talk about prioritizing tests for automation, it often comes down to high repetitiveness of actions and high number of iterations. Data-driven testing fits that pattern well and helps with some of the adoption barriers:

  1. It removes the need to create an individual script for each test case, which is typically a time-consuming process. But often more importantly, it simplifies and accelerates maintenance because most changes to the test script do not affect the test data, and vice versa. Scaling execution up or down is also more flexible.
     

    As an added bonus, if you need to switch frameworks/languages, your data sources would still be valid, and not as many scripts would need to be replaced.
     

  2. It can make collaboration easier since reviewing a data table does not require special technical skills. That, in turn, accelerates the overall process (since any subject matter expert can add iterations) and increases test data accuracy. The reporting can be set up to clearly reflect the status of each iteration, just as it would for separate scripts.

Of course, data-driven automation is not the universal solution, a few of its limitations are:

  1. Skill level requirements - if each iteration has enough “uniqueness” about its actions & validations or complexity of the expected result, trying to fit numerous iterations into 1 script will result in the overload of conditional statements. That would require a lot more automation experience.
     
  2. Shifting some of the problems to a different medium - manual creation & maintenance of data sources can be only marginally less cumbersome than those of individual scripts. Data relevancy and accuracy can still be large concerns.

For the first limitation, the collaborative analysis of the draft data tables should validate the pattern early on and properly choose the targets for data-driven automation, without excessively complicating the overall implementation. For the avoidance of doubt, you will likely always have a mix of automation approaches, where data-driven testing is one part.

For the second one, we can find a better alternative to manual data source management. For example, DesignWise. 

Why data-driven automation with DesignWise can be an even better choice

  • Faster data source creation - designing the table by hand can’t compete with the algorithmic generation when it comes to speed, especially for complex systems.
     
  • More efficient maintenance - changes at the model level are automatically applied to all affected test cases.
     
  • Minimized redundancy & maximized thoroughness - the DesignWise algorithm makes sure the data scenarios are as varied as possible while providing flexible coverage goals. That often leads to earlier detection of defects.
     
  • Diverse collaboration artifacts - in addition to the data table, you can leverage auto-generated visualizations like mind map and coverage matrix to make sure all stakeholders are on the same page about the testing scope.

If at any point you need to switch to the “traditional” script-based approach, you can easily do it by leveraging the data-driven BDD/Robot Framework templates under Scripts. Create 1 Scenario/Test Case block inside DesignWise, export it to as many tests as you have iterations. You don’t need to rebuild your models, and you still benefit from all 4 points above.

So let’s dive into how such optimized data-driven automation can be accomplished.
 

Prerequisites

For Playwright, in order to run this tutorial you need to have Nodejs and install the playwright-test runner.

For Cypress, you need to have Cypress installed.

 

How to do it

Common steps

We will discuss the implementation in 2 frameworks, with different complexity of the target apps. At a high level, steps are similar for each example:

  1. Create a model for the target system in DesignWise.
  2. Export the data table as CSV.
  3. Create the data-driven execution script in the selected framework/language.

  4. Run that code and generate the report format that supports iteration-level results.

     

Integrate DesignWise with Playwright

For Playwright versions 1.34 and later, if you want to attach screenshots, you will need to additionally install this Playwright reporter and change the config file or the command line statement accordingly. 

The scope of this tutorial is:

  • passing different combinations of data to the calculator app at https://treasurydirect.gov/BC/SBCGrw
  • validating that the evaluation results are returned successfully (by checking the presence of the title on the page, since we don't know the actual formula behind the app)

 

Build a DesignWise model and export scenarios

The model in DesignWise looks like this (a fairly straightforward approach of a parameter per UI field):

Note that:

  • Shorter parameter names without spaces or special characters work better.
  • The spelling of value names should match the syntax expected by the automation script. You can share your draft DesignWise model with the engineers to verify that.
  • For data table and Scripts->Automate exports, only value expansion text will be populated, so we can still benefit from the equivalence class approach for, e.g., IntRate or FedTaxRate, without worrying about the value name syntax.

After generating Scenarios, we can export the CSV data table from the same screen:

 

As the automation is being developed, it may make sense to leverage the “1-way interactions” goal for the DesignWise algorithm. It exposes the execution to each possible value with the smallest number of iterations, which helps with the feedback speed.

 

Create a script in Playwright

On the Playwright side, you may first need to adjust the configuration file to support the appropriate iteration-based report formatting (we use JUnit reporter + Xray options plugin in this tutorial). Then we implement the driver script:

DWDemo.spec.ts

import { test, expect } from '@playwright/test';
import fs from 'fs';
import path from 'path';
import { parse } from 'csv-parse/sync';


const records = parse(fs.readFileSync(path.join(__dirname, 'DWDemo.csv')), {
	columns: true,
	skip_empty_lines: true
});


for (const record of records) {
  test(`${record.Summary}`, async ({ page }, testInfo) => {
	testInfo.annotations.push({ type: 'requirements', description: 'TAC-117' });
	testInfo.annotations.push({ type: 'test_description', description: `Confirm the calculation is successful for Initial Amount = ${record.InitialInvAmount}, IntRate = ${record.IntRate}, Periodic Investment = ${record.PeriodicInv},
	Frequency = ${record.Frequency}, Years Invested = ${record.YearsInvested}, Federal Tax Rate = ${record.FedTaxRate} `});
	
	console.log(record.Summary, record.InitialInvAmount, record.IntRate, record.PeriodicInv, record.Frequency, record.YearsInvested, record.FedTaxRate);


	await page.goto('https://treasurydirect.gov/BC/SBCGrw');
	await expect(page.getByRole('heading', { name: 'Growth Calculator' })).toBeVisible();
	
	await page.locator('input[name="InitInvest"]').fill(record.InitialInvAmount);
	await page.locator('select[name="IntRate"]').selectOption({label:record.IntRate});
	await page.locator('select[name="PeriodicPmt"]').selectOption({label:record.PeriodicInv});
	await page.locator('select[name="PmtFrequency"]').selectOption({label:record.Frequency});
	await page.locator('input[name="Years"]').fill(record.YearsInvested);
	await page.locator('input[name="FedTax"]').fill(record.FedTaxRate);
	
	const completeinputs = await page.screenshot();
	await page.getByRole('button', { name: 'CALCULATE' }).click();
	await expect(page.getByRole('heading', { name: 'Growth Calculator Results' })).toBeVisible();
	const calcresults = await page.screenshot();
	await page.getByRole('button', { name: 'Back to Growth Calculator' }).click();
	
	await testInfo.attach('completeinputs', { body: completeinputs, contentType: 'image/png' });
	await testInfo.attach('calcresults', { body: calcresults, contentType: 'image/png' });


  });
};

The core detail about this approach is, using the extra libraries, we first read the csv from DesignWise, then iterate through it with the “for” loop:

CSV Handling and Parameterized Steps

...
import fs from 'fs';
import path from 'path';
import { parse } from 'csv-parse/sync';


const records = parse(fs.readFileSync(path.join(__dirname, 'DWDemo.csv')), {
	columns: true,
	skip_empty_lines: true
});


for (const record of records) {
  ...
};

The parameters (aka column names from the CSV, e.g., record.InitialInvAmount) are passed in typical Playwright methods (like Fill or selectOption) instead of hardcoding text.

 ...
	await page.locator('input[name="InitInvest"]').fill(record.InitialInvAmount);
	await page.locator('select[name="IntRate"]').selectOption({label:record.IntRate});
	await page.locator('select[name="PeriodicPmt"]').selectOption({label:record.PeriodicInv});
	await page.locator('select[name="PmtFrequency"]').selectOption({label:record.Frequency});
	await page.locator('input[name="Years"]').fill(record.YearsInvested);
	await page.locator('input[name="FedTax"]').fill(record.FedTaxRate);
...

Optional Customization

To improve the clarity and traceability, we can add a link to the requirement story, parameterized name & description, and custom screenshots.

...
  test(`${record.Summary}`, async ({ page }, testInfo) => {
	testInfo.annotations.push({ type: 'requirements', description: 'TAC-117' });
	testInfo.annotations.push({ type: 'test_description', description: `Confirm the calculation is successful for Initial Amount = ${record.InitialInvAmount}, IntRate = ${record.IntRate}, Periodic Investment = ${record.PeriodicInv},
	Frequency = ${record.Frequency}, Years Invested = ${record.YearsInvested}, Federal Tax Rate = ${record.FedTaxRate} `});


...	
	
	const completeinputs = await page.screenshot();
	const calcresults = await page.screenshot();
...	
	await testInfo.attach('completeinputs', { body: completeinputs, contentType: 'image/png' });
	await testInfo.attach('calcresults', { body: calcresults, contentType: 'image/png' });

Integrate DesignWise with Cypress

The scope of this tutorial is:

  • Providing different combinations of applicant data to the web mortgage quoting system at https://www.wellsfargo.com/mortgage/calculators/. 
    • The flow starts with selecting “Buy a home” as the purpose, then entering location, amount, and credit rating details. After the initial results, you can adjust the term.

  • Checking the app's ability to return the loan evaluation results successfully by asserting the page title (without looking at the specific numbers).

     

Build a DesignWise model and export scenarios

The model in DesignWise looks like this:

Note that:

  • It’s not wrong to include 1-value parameters for test data rationale (like “Purpose”) if you feel like it’s better than hardcoding them in the script (often because you expect that parameter to grow over time).
  • For some web apps, selection by labels can be too complicated, so you may have to switch to indices (e.g. “Term” values). Refer to the feedback from your automation team.
  • This example showcases how to deal with conditional steps.

    “If the DownpaymentPercentage is 20 or higher, then an additional question about the number of borrowers doesn’t appear". On the DesignWise side, you can have:
    • either a “special” value (like “NABorrowers” for “NumberofPeople”) with the corresponding set of constraints

    • or a “skip” constraint (DownpaymentPercentage[20,21 - 50] >> NumberofPeople)

      The choice will determine which trigger will be assigned to the automation step that we will show later.

       

Create a script in Cypress

On the Cypress side, you will also first need to adjust the package and the config files to support csv handling with 3 imports:

Package.json

{
  "type": "module",
  "devDependencies": {
    "cypress": "^12.9.0",
    "cypress-if": "^1.10.4",
    "mocha": "^10.2.0",
    "mochawesome": "^7.1.3",
    "mochawesome-merge": "^4.3.0",
    "mochawesome-report-generator": "^6.2.0"
  },
  "dependencies": {
    "cypress-mochawesome-reporter": "^3.3.0",
    "neat-csv": "^7.0.0",
    "node-fetch": "^2.6.1"
  }
}

cypress.config.js

import { defineConfig } from 'cypress'


import fs from 'fs'
import path from 'path'
import neatCSV from 'neat-csv'


export default defineConfig({
  e2e: {
	async setupNodeEvents (on, config) {
		const filename = './cypress/fixtures/DWDemo.csv'
		console.log('loading file', filename)
      	const text = fs.readFileSync(filename, 'utf8')
      	const csv = await neatCSV(text)
      	console.log('loaded the quote data')
      	console.log(csv)


		config.env.DWDemo = csv
		return config
    },
  },
})

The optimized data table can go into cypress/fixtures, while the execution script in cypress/e2e would look like this:

DWDemoScript.cy.js

const dwloandata = Cypress.env('DWDemo')
	describe(`Quote a mortgage`, () => {
	    dwloandata.forEach((l) => {
			it(`With ${l.PurchasePrice} purchase price and ${l.DownpaymentPercentage} downpayment for ${l.NumberofPeople}`, () => {
				cy.visit('https://www.wellsfargo.com/mortgage/calculators/')
				cy.contains(l.Purpose).click()
				cy.get('#label-locationField').click()
				cy.get('#locationField').type(l.Location)
				cy.contains('Next').click()
				cy.get('#label-purchasePriceAmount').click()
				cy.get('#purchasePriceAmount').type(l.PurchasePrice)
// the location suggestions window sometimes persists through all the subsequent steps, so added {force: true} to ignore it
				cy.contains('Next').click({force: true})
				cy.get('#downPaymentPercent').clear()
				cy.get('#downPaymentPercent').type(l.DownpaymentPercentage)
				cy.contains('Next').click({force: true})
				
				cy.get("body").then($body => {
					if ($body.find(`#${l.NumberofPeople}`).length > 0) {   
					//evaluates as true if button exists at all
						cy.get(`#${l.NumberofPeople}`).then($btn => {
							if ($btn.is(':visible')){
							//you get here only if button EXISTS and is VISIBLE
							cy.wrap($btn).click({force: true})
							} else {
							//you get here only if button EXISTS but is INVISIBLE
							cy.log('Question about borrowers did not appear')
							}
							});
						}	else {
							//you get here if the button DOESN'T EXIST
							cy.log('Question about borrowers did not exist on the page')
							assert.isOk('everything','everything is OK');
							} 
					})
				
				cy.get(`#${l.creditRating}`).click({force: true})
				
				// form
				cy.get('#label-aboutYourselfFirstName').click()
				cy.get('#aboutYourselfFirstName').type('I')
				cy.get('#label-aboutYourselfLastName').click()
				cy.get('#aboutYourselfLastName').type('F')
				cy.get('#label-aboutYourselfEmail').click()
				cy.get('#aboutYourselfEmail').type('test@test.org')
				cy.get('#label-aboutYourselfPhone').click()
				cy.get('#aboutYourselfPhone').type('1111111111')
				cy.get('#label-aboutYourselfZipCode').click()
				cy.get('#aboutYourselfZipCode').type('28262')
				
				cy.get('#continue-view-rates').click({force: true})
				cy.get('#consent-submit-click').click({force: true})
				
				cy.get('#RST_Results_LoanTerm').select(l.Term)
				 
				cy.get('#skip').should('have.text', '\n\t\t\t\tCustomized Mortgage Rate Results\n\t\t\t')
				
			})
		})
	})

Similarly to the Playwright example, we will first read the CSV file, then iterate the script through it with the “forEach” loop.

DWDemoScript.cy.js

const dwloandata = Cypress.env('DWDemo')
	...
	    dwloandata.forEach((l) => {
			...
	})

Keep in mind that the final naming convention for the test in the report will often concatenate these lines:

DWDemoScript.cy.js

describe(`Quote a mortgage`, () => {
	   ...
			it(`With ${l.PurchasePrice} purchase price and ${l.DownpaymentPercentage} downpayment for ${l.NumberofPeople}`, () => {
				...
		})
	})

So you may need to split the desired summary accordingly while parameterizing the second line.

 

The code below is responsible for the conditional logic we mentioned earlier around Downpayment Percentage and Number of Borrowers.

DWDemoScript.cy.js

...			
				cy.get("body").then($body => {
					if ($body.find(`#${l.NumberofPeople}`).length > 0) {   
						cy.get(`#${l.NumberofPeople}`).then($btn => {
							if ($btn.is(':visible')){
							cy.wrap($btn).click({force: true})
							} else {
							cy.log('Question about borrowers did not appear')
							}
							});
						}	else {
							cy.log('Question about borrowers did not exist on the page')
							assert.isOk('everything','everything is OK');
							} 
					})
				
			...

It searches for the dynamically defined element (`#${loan.NumberofPeople}`), checks whether it’s present and visible, and, if yes, clicks it, otherwise, logs a message. In this implementation of the DesignWise model, ‘#NABorrowers’ element would never be found, so the automation skips the step whenever that value comes from the CSV.

- If you know the exact behavior of the element (visibility vs presence), that conditional logic can be simplified.
- If the “skip” constraint route is taken, the same code could work. Or you may need to adjust the dynamic element syntax if ‘#_‘ is treated as undefined/syntax error.

At the time of the article, 2 tests in this suite were failing as the results were not returned. This is a decent example of the execution feedback loop for combinatorial test design: we will often exercise the combinations that haven't been explicitly mentioned in requirements. Since all individual values are within valid ranges (based on UI tooltips), we would review the scenario as a whole with the stakeholders and confirm the expected behavior. (Both failures shared at least 1 similarity: the purchase price was close to the upper limit.)

If the lack of results is valid, we can make the assertion conditional on data in each iteration or adjust the values/constraints in the DesignWise model to focus more on the combinations that return rates.

 

Tips

  • Generating CSV from the Scenarios screen in DesignWise results in only value expansion names being included, where applicable. The CSV option on the Export screen is geared more towards manual execution and will produce "Value name - Value expansion name" syntax.
  • To create script steps in Playwright, if you are not yet familiar with the syntax, you can use the CodeGen feature.
  • The csv processing for Cypress recommends neat-csv package. Depending on the version, you need to switch the Cypress config and package files between CommonJS and ES Module scripting. Our example uses ES Module. Or you can try an alternative package.

 

References

 

Was this article helpful?
Powered by Zendesk