[Feature] advanced configurations for routeFromHAR

[NoamGaash]

Background / Use cases

The routeFromHAR functionality encapsulates two behaviors. When update is true, it behaves as a custom network recorder, while when update is false (default) it serves as a network communication mock.
I would like to suggest several improvements:

saving minimal data

today, each HAR entry contains a lot of unnecessary data - timing, HTTP version, request headers, and more.
while this data might be useful for analysis, most of it is not being used for network traffic replay.
Despite that omitting the unnecessary fields would make a deviation from the formal HAR specifications, I believe it is worth it in terms of clarity, bundle size, concise git differences, output predictability, and maintainability.

example

before:

      {
        "startedDateTime": "2023-03-05T05:09:17.557Z",
        "time": 8.897,
        "request": {
          "method": "GET",
          "url": "http://localhost:10102/api/pages?accountId=<my_id>",
          "httpVersion": "HTTP/1.1",
          "cookies": [],
          "headers": [
            { "name": "Accept", "value": "application/json, text/plain, */*" },
            { "name": "Accept-Language", "value": "en-US" },
            { "name": "Cookie", "value": "cookies....." },
            { "name": "Expect", "value": "202+location" },
            { "name": "Referer", "value": "http://localhost:port/path?accountId=id },
            { "name": "User-Agent", "value": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/110.0.5481.177 Safari/537.36" },
            { "name": "sec-ch-ua", "value": "" },
            { "name": "sec-ch-ua-mobile", "value": "?0" },
            { "name": "sec-ch-ua-platform", "value": "" },
            { "name": "x-client-request-id", "value": "8a887b6a-7638-4e0a-bf04-b4c97a78199e--4" }
          ],
          "queryString": [
            {
              "name": "accountId",
              "value": "<my id>"
            }
          ],
          "headersSize": -1,
          "bodySize": -1
        },
        "response": {
          "status": 200,
          "statusText": "OK",
          "httpVersion": "HTTP/1.1",
          "cookies": [],
          "headers": [
            { "name": "X-Powered-By", "value": "Express" },
            { "name": "cache-control", "value": "no-store" },
            { "name": "connection", "value": "close" },
            { "name": "content-encoding", "value": "gzip" },
            { "name": "content-type", "value": "application/json; charset=utf-8" },
            { "name": "date", "value": "Sun, 05 Mar 2023 05:09:17 GMT" },
            { "name": "strict-transport-security", "value": "max-age=31536000" },
            { "name": "transfer-encoding", "value": "chunked" },
            { "name": "vary", "value": "Accept-Encoding" },
            { "name": "x-content-type-options", "value": "nosniff" },
            { "name": "x-permitted-cross-domain-policies", "value": "None" },
            { "name": "x-xss-protection", "value": "1; mode=block" }
          ],
          "content": {
            "size": -1,
            "mimeType": "application/json; charset=utf-8",
            "text": "{\"pages\":[{\"id\":\"1234\",\"name\":\"a new created test project\",\"createdAt\":\"2023-03-05T05:09:18.0612396+00:00\",\"updatedAt\":\"2023-03-05T05:09:18.0612396+00:00\",\"source\":\"filesystem\",\"status\":\"New\",\"metadata\":{}},{\"id\":\"00000251727294129115\",\"name\":\"with implementation\",\"createdAt\":\"2023-01-29T15:37:50.8846963+00:00\",\"updatedAt\":\"2023-01-29T15:38:07.5206989+00:00\",\"source\":\"filesystem\",\"status\":\"None\",\"metadata\":{}},{\"id\":\"00000251727302447488\",\"name\":\"no implementation\",\"createdAt\":\"2023-01-29T13:19:12.5111176+00:00\",\"updatedAt\":\"2023-01-29T15:37:23.8938755+00:00\",\"source\":\"filesystem\",\"status\":\"None\",\"metadata\":{}}]}"
          },
          "headersSize": -1,
          "bodySize": -1,
          "redirectURL": ""
        },
        "cache": {},
        "timings": { "send": -1, "wait": -1, "receive": 8.897 }
      }

after:

      {
        "request": {
          "method": "GET",
          "url": "http://localhost:10102/api/pages?accountId=<my_id>"
          ]
        },
        "response": {
          "status": 200,
          "statusText": "OK",
          "httpVersion": "HTTP/1.1",
          "cookies": [],
          "headers": [
            { "name": "X-Powered-By", "value": "Express" },
            { "name": "cache-control", "value": "no-store" },
            { "name": "connection", "value": "close" },
            { "name": "content-encoding", "value": "gzip" },
            { "name": "content-type", "value": "application/json; charset=utf-8" },
            { "name": "date", "value": "Sun, 05 Mar 2023 05:09:17 GMT" },
            { "name": "strict-transport-security", "value": "max-age=31536000" },
            { "name": "transfer-encoding", "value": "chunked" },
            { "name": "vary", "value": "Accept-Encoding" },
            { "name": "x-content-type-options", "value": "nosniff" },
            { "name": "x-permitted-cross-domain-policies", "value": "None" },
            { "name": "x-xss-protection", "value": "1; mode=block" }
          ],
          "content": {
            "mimeType": "application/json; charset=utf-8",
            "text": "{\"pages\":[{\"id\":\"1234\",\"name\":\"a new created test project\",\"createdAt\":\"2023-03-05T05:09:18.0612396+00:00\",\"updatedAt\":\"2023-03-05T05:09:18.0612396+00:00\",\"source\":\"filesystem\",\"status\":\"New\",\"metadata\":{}},{\"id\":\"00000251727294129115\",\"name\":\"with implementation\",\"createdAt\":\"2023-01-29T15:37:50.8846963+00:00\",\"updatedAt\":\"2023-01-29T15:38:07.5206989+00:00\",\"source\":\"filesystem\",\"status\":\"None\",\"metadata\":{}},{\"id\":\"00000251727302447488\",\"name\":\"no implementation\",\"createdAt\":\"2023-01-29T13:19:12.5111176+00:00\",\"updatedAt\":\"2023-01-29T15:37:23.8938755+00:00\",\"source\":\"filesystem\",\"status\":\"None\",\"metadata\":{}}]}"
          }
        }
      }

excludeUrl

today, the page.routeFromHAR method receives URL option that defines which URLs should be included in the resulting HAR file.
I suggest we should have excludeUrl property. the property would allow users to exclude specific path(s) from recording.

example

before:

page.routeFromHAR(
        `__tests__/networks_cache/${step}.har`,
        {
            url: /api|css|png|tff/, // user tries to list all URLs that could be memorized in the HAR file
        }
    );

after

page.routeFromHAR(
        `__tests__/networks_cache/${step}.har`,
        {
            url: /.*/, // user can use a wildcard
            excludeUrl: "http://localhost/bundle.js" // user exclude specific path
        }
    );

smart matching algorithm

Playwright docs state that:

HAR replay matches URL and HTTP method strictly. For POST requests, it also matches POST payloads strictly. If multiple recordings match a request, the one with the most matching headers is picked

Would it be possible to grant users more control over he match algorithm?

example

page.routeFromHAR(
        `__tests__/networks_cache/${step}.har`,
        {
            url: /api/,
            urlMatching: {
                "GET": {
                    matchLocation: "ignoreOrigin",
                    matchHeaders: false,
                    
                },
                "POST": {
                    matchPayload: false
                },
                "delete": {
                    match: (harEntry, currRequest) => {
                        return userDefinedMatchScoringFunction(harEntry, currRequest);
                    }
                }
            }
        }
    );

summery

Thank you for considering my suggestions!
Please note that my first suggestion is a breaking change - it would omit data from the HAR file, and there is a chance that some users rely on the timing/headers data in there.
I would love to hear your opinion and get some feedback before implementing any changes or submitting PRs.

Thank you for maintaining Playwright! I love this tool.
Noam

[fs-projects]

Hi All,

I think this is the most appropriate place I could think of to post one of my query.

I recently setup a HAR in my tests that captures all network requests in a HAR file when update flag is true and then use this HAR file to serve the mock API responses when update flag is false. Few things I observed that are kind of blocking me to actually leverage HAR functionality. Please see below -

1. There is an API call that takes sometime to get the results back. It's output is quite huge and when I see the preview tab in network section of chrome, I see this message Failed to load response data: Request content was evicted from inspector cache. Although UI was able to get all the data and display the results but I am not able to see any data in preview tab in network section of chrome. Logically suspecting this is causing playwright to NOT record the response of the API as well in HAR file. In HAR file there is no _file key for this particular network call.

Workaround : To fix this I manually added _file key and assigned a value test.json to it. I created test.json in my har directory and added the response of this API. How I got the response? I called this API in postman to get the full response.

2. Now applying above workaround I ran everything again. Then there is another problem I encountered. There are some urls in some API calls that passes a query parameter start to backend. It's an epoch time value. When I record HAR with update flag as true, start has certain value in it corresponding to the epoch time when API was called. When I use the same HAR file with update flag as false then the same API url doesn't match with the one in HAR file as the new url is different from what was saved in HAR file because start parameter has a different value now.

www.example.com/api/v2/collect/?start=1695901908803 -- Saved in HAR file when recorded(when update flag true)
www.example.com/api/v2/collect/?start=1695901912345 -- Saved when test is run from HAR(when update flag false) then application calls the same API but with different value of start time. Due to this when playwright mocks this API call it doesn't find the exact match in HAR file and the mocking fails by default.

For point 2 I wanted to know if there is anything I can do ensure that even though the there is a change in start parameter playwright mocks the API because for testing purpose I don't care about the result difference between the two timestamps. I just want playwright to use the same mock again.

One lengthy method is that I mock such Apis separately with glob pattern and fulfill it with the whatever result I want. But doing so I won't be able to leverage the HAR power and this might become cumbersome for other pages also where such type of APIs are called.

Any thoughts/guidance highly appreciated.. Thanks!!

[NoamGaash]

@fs-projects in my projects I use things like:

  await page.routeFromHAR( /* ...........  */)
  await page.route(/v2\/collect/, (route) => {
    const url = route.request().url().replace(/\d+/, "1695901908803")
    route.fallback({url})
  })

But I feel terrible doing it. its not maintainable at all.
I'm trying to implement the smart matching algorithm into Playwright, but I'm facing some technical difficulties regarding the protocol.yml file.
If anyone want's to join my efforts, I would love to have an online meeting and get some feedback
I'm available on noam.gaash@applitools.com

[fs-projects]

@NoamGaash I was on leave for some days. Thanks for sharing this workaround. I tweaked it as per my app and it worked. I know it's not maintainable but great help. Could you please let me know if my understanding is correct -

await page.routeFromHAR( /* ........... */) -- Will serve the request from HAR file instead of actual network calls.

    const url = route.request().url().replace(/\d+/, "1695901908803")
    route.fallback({url})
  }) 

--Ensures all matching urls are replaced as specified and then searched and served from the HAR file? Let me know in detail if possible.

Also I am happy to connect with you to join your efforts. I will shoot out an email to you and you can let me know if we can proceed.

[NoamGaash]

sure! thanks.

Playwright interceptors (page.route, page.routeFromHar) are implemented with a stack mechanism - the first route function you register will be the last to handle the request.
for example, consider the following URL:
www.example.com/api/v2/collect/?start=1695901912345
and the following code:

  await page.routeFromHAR( /* ...........  */)
  await page.route(/v2\/collect/, (route) => {
    const url = route.request().url().replace(/\d+/, "1234")
    route.fallback({url})
  })

1. first, the URL will meet the last page.route.
2. the route.fallback will make a request to www.example.com/api/v2/collect/?start=1234 to proceed to the next interceptor
3. the new (constant) URL will meet page.routeFromHAR
4. everyone is happy (except us the programmers)

[NoamGaash]

Hi everyone!
I've made a small npm-package, that solve that issue.
With the playwright-advanced-har package, you'll be able to:

• Ignore port numbers
• Ignore search params
• Shuffle response order

and much more

please don't hesitate to open an issue with any question, and I'll do my best to help any use case

@fs-projects
your use case can be solved using:

import { test, defaultMatcher } from "playwright-advanced-har";

test("ignore search params", async ({ page, advancedRouteFromHAR }) => {
	await advancedRouteFromHAR("tests/har/different-search-params.har", {
		matcher: (request, entry) => {
			const reqUrl = new URL(request.url());
			const entryUrl = new URL(entry.request.url);
			reqUrl.search = "";
			entryUrl.search = "";
			if (
				reqUrl.toString() === entryUrl.toString() &&
				request.method() === entry.request.method &&
				request.postData() == entry.request.postData?.text
			) {
				return 1;
			}
			return -1;
		},
	});
	await page.goto("www.example.com/api/v2/collect/?start=" + Date.now());
});

[bcowgill]

I've just had a difficulty with routeFromHAR where we have a cache busting parameter in our API url so the har file does not match.

/api?a=3648574675856 for example where a changes on every api call.

I have basically made my own version of routeFromHAR to match the URL and Method but pass in a santitseUrl function which is called on the route.request().url() and the har.log.entries[].request.url values before matching them.

sanitiseUrl(url) url.replace(/([?&]a=)\d+/. "$1NNNNNNNNN")

You might consider adding such a parameter here to advanced options if you are planning improvements.

[NoamGaash]

@bcowgill I believe this snippet will solve your use case:

import { test, customMatcher } from "playwright-advanced-har";

const fixUrl = url => url.replace(/([?&]a=)\d+/. "$1NNNNNNNNN")
test("ignore `a` get argument", async ({ page, advancedRouteFromHAR }) => {
	await advancedRouteFromHAR("tests/har/my-file.har", {
		matcher: customMatcher({
			urlComparator(a, b) {
				return fixUrl(a) === fixUrl(b);
			},
		}),
	});
	await page.goto("/api?a=3648574675856");
});

[bcowgill]

@bcowgill I believe this snippet will solve your use case:

import { test, customMatcher } from "playwright-advanced-har";

Hey, thanks, how about this other issue, will it solve that?

routeFromHAR header Access-Control-Allow-Origin should be configurable
need to replace test env domain with localhost domain to play back captures locally
#28447

[NoamGaash]

@bcowgill actually, I'm considering adding that exact feature
https://github.com/NoamGaash/playwright-advanced-har/pull/6/files
It will let you alter the entry found in the HAR.
WDYT?
Please open an issue for that, I have several ideas but I want to be backward compatible as much as possible, therefore I release new features very cautiously

[fs-projects]

Hi everyone! I've made a small npm-package, that solve that issue. With the playwright-advanced-har package, you'll be able to:

• Ignore port numbers
• Ignore search params
• Shuffle response order

and much more

please don't hesitate to open an issue with any question, and I'll do my best to help any use case

@fs-projects your use case can be solved using:

import { test, defaultMatcher } from "playwright-advanced-har";

test("ignore search params", async ({ page, advancedRouteFromHAR }) => {
	await advancedRouteFromHAR("tests/har/different-search-params.har", {
		matcher: (request, entry) => {
			const reqUrl = new URL(request.url());
			const entryUrl = new URL(entry.request.url);
			reqUrl.search = "";
			entryUrl.search = "";
			if (
				reqUrl.toString() === entryUrl.toString() &&
				request.method() === entry.request.method &&
				request.postData() == entry.request.postData?.text
			) {
				return 1;
			}
			return -1;
		},
	});
	await page.goto("www.example.com/api/v2/collect/?start=" + Date.now());
});

Thank you very much @NoamGaash

[giladgd]

Is there any update about this?
When will it be part of Playwright?