Add gzip compression for URL metrics using Compression Streams API #1959

b1ink0 · 2025-03-27T17:53:22Z

Summary

Relevant technical choices

This PR implements the Compression Streams API to efficiently compress URL Metrics JSON before sending it. This significantly reduces the payload size, allowing more data to be sent without exceeding the 64 KiB limit.

Previously some of these changes where reverted due to the bug #1928.

This PR is able to avoid the issue of URL metrics not being sent on pagehide event by compressing the URL metrics before pagehide event and then recompressing it whenever URL metrics are updated by the extensions.

…etric-gzip-compression" This reverts commit aef2e8d, reversing changes made to 47eef00.

Co-authored-by: Felix Arntz <[email protected]>

codecov · 2025-03-27T18:04:02Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 72.51%. Comparing base (583a4e9) to head (6aa261c).
Report is 27 commits behind head on trunk.

Additional details and impacted files

@@            Coverage Diff             @@
##            trunk    #1959      +/-   ##
==========================================
+ Coverage   72.34%   72.51%   +0.17%     
==========================================
  Files          85       85              
  Lines        6979     7023      +44     
==========================================
+ Hits         5049     5093      +44     
  Misses       1930     1930

Flag	Coverage Δ
multisite	`72.51% <100.00%> (+0.17%)`	⬆️
single	`40.33% <0.00%> (-0.26%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

b1ink0 · 2025-03-27T18:10:49Z

If we do this, we should add a pagehide event handler before we load any of the extensions, so that our pagehide callback is called first, which then causes extendElementData and extendRootData to no-op (or throw an error) because at this point we can't re-compress the URL Metric since it is too late.

#1893 (comment)

I would like to understand more about adding a pagehide event handler before loading any extensions. From my interpretation, the goal is to override the extendElementData and extendRootData functions to throw an error if an extension tries to modify URL metrics after the pagehide event.

However, we're currently still supporting the finalize method, even though it's deprecated. Introducing this change might cause issues for extensions that still rely on finalize.

At the moment, I’ve added a flag called enableCompression, which is set to false if a finalize function is detected. This effectively disables compression. I could use this same flag to prevent the function overrides that would otherwise throw errors.

cc: @westonruter

westonruter · 2025-03-27T18:15:06Z

plugins/optimization-detective/detect.js

+/**
+ * Timeout ID for debouncing URL metric compression.
+ *
+ * @type {ReturnType<typeof setTimeout> | null}


Is this any different than the following?

Suggested change

* @type {ReturnType<typeof setTimeout> | null}

* @type {number|null}

Yes its different TypeScript throws as error if we only put number

Type 'Timeout' is not assignable to type 'number'

recompressionTimeout = setTimeout( async () => {

Ah, good to know.

plugins/optimization-detective/detect.js

westonruter · 2025-03-27T18:22:11Z

plugins/optimization-detective/detect.js

@@ -660,6 +724,10 @@ export default async function detect( {
 					initializingExtensionModuleUrls.push( extensionModuleUrl );
 				}
 			}
+
+			if ( extension.finalize instanceof Function ) {
+				compressionEnabled = false;


Maybe better to have a separate variable like extensionHasFinalize which is initially false but then is set to true here. Then after the loop finishes, if extensionHasFinalize is true, it can at that point set compressionEnabled to false and at the same time emit a warning to say that URL Metric compression is disabled due to the use of the deprecated finalize function.

Done in 7e8ec60.

westonruter · 2025-03-27T18:24:54Z

plugins/optimization-detective/detect.js

+	recompressionTimeout = setTimeout( async () => {
+		if ( typeof requestIdleCallback === 'function' ) {
+			await new Promise( ( resolve ) => {
+				requestIdleCallback( resolve );


The requestIdleCallback() returns a handle value which can be passed to cancelIdleCallback(). I think this should be done as well in addition to clearing the debouncing timeout, as otherwise it could happen that two idle callbacks for re-compressing the URL Metric could happen right after each other.

Done in aeda409.

westonruter · 2025-03-27T18:27:06Z

I would like to understand more about adding a pagehide event handler before loading any extensions. From my interpretation, the goal is to override the extendElementData and extendRootData functions to throw an error if an extension tries to modify URL metrics after the pagehide event.

@b1ink0 I think that comment I made is obsolete. We should prevent compression from happening if there is a finalize function exported from an extension, like you're doing in the PR.

Co-authored-by: Weston Ruter <[email protected]>

…tions Co-authored-by: Weston Ruter <[email protected]>

…alize` function Co-authored-by: Weston Ruter <[email protected]>

plugins/optimization-detective/detect.js

plugins/optimization-detective/detection.php

westonruter · 2025-03-31T01:31:00Z

plugins/optimization-detective/detect.js

+			} );
+			idleCallbackHandle = null;
+		}
+		compressedPayload = await compress( JSON.stringify( urlMetric ) );


Just in case this throws an error, perhaps there should be a try/catch block around this which when an error is caught the error is logged out to the console and then compressionEnabled is set to false.

For logging the error, should console.log be used since debouncedCompressUrlMetric function is defined outside the detect function and therefore doesn't have access to the custom logger? Also, if we want to use the custom logger, then we would need to pass it through multiple functions, as debouncedCompressUrlMetric is called inside both extendElementData and extendRootData. Another option could be to define the logger constant as a let outside the detect function which could let us use the custom logger.

The function could also just create a new logger which is only used here for logging out the error.

Another option would be to re-throw the error after setting compressionEnabled to false. But better to log it.

The function could also just create a new logger which is only used here for logging out the error.

So should the debugMode param should be set to true regardless of the isDebug passed to detect function? One other thing creating a logger for each call to debouncedCompressUrlMetric does not seem correct to me, should there be a new variable outside function like compressionLogger?

The logger would only be created in the catch block and then it would never be created again since upon an error compressionEnabled would be set to false and the function would short-circuit. So that seems fine.

And the debugMode param shouldn't matter since the error function logs out unconditionally.

Ohh, It makes sense thank you!

Done in 6c8c257

Co-authored-by: Weston Ruter <[email protected]>

westonruter · 2025-03-31T05:41:37Z

plugins/optimization-detective/detect.js

+			compressionEnabled = false;
+		}
+		recompressionTimeout = null;
+	}, 1000 );


Minor point: let's put this 1000 in a constant defined up near the top of this module so we can more easily understand what the magic number is. For example, compressionDebounceWaitDuration.

Done in dc4d5e9

Co-authored-by: Weston Ruter <[email protected]>

github-actions · 2025-03-31T06:39:57Z

The following accounts have interacted with this PR and/or linked issues. I will continue to update these lists as activity occurs. You can also manually ask me to refresh this list by adding the props-bot label.

If you're merging code through a pull request on GitHub, copy and paste the following into the bottom of the merge commit message.

Co-authored-by: b1ink0 <[email protected]>
Co-authored-by: westonruter <[email protected]>
Co-authored-by: phanduynam <[email protected]>

To understand the WordPress project's expectations around crediting contributors, please review the Contributor Attribution page in the Core Handbook.

…begin with

… add/reintroduce-url-metrics-gzip-compression

westonruter

Thanks so much for bringing this back to life!

westonruter · 2025-04-03T22:03:49Z

plugins/optimization-detective/detect.js

@@ -718,12 +908,16 @@ export default async function detect( {
 		urlMetric.elements.push( elementData );
 		elementsByXPath.set( elementData.xpath, elementData );
 	}
+	breadcrumbedElementsMap.clear();


Oops. This duplicate call (which I added) I've removed in #1968

b1ink0 and others added 8 commits March 26, 2025 15:05

Revert "Merge pull request WordPress#1929 from WordPress/revert/url-m…

0128b8d

…etric-gzip-compression" This reverts commit aef2e8d, reversing changes made to 47eef00.

Use Content-Encoding:gzip instead of Content-Type:application/gzip

12a5a56

Move up breadcrumbedElementsMap.clear() to logical spot

8e8d8f8

Check if gzdecode() exists first

3ebafc7

Add filter to control whether gzip is used for URL Metric payloads

db48f56

Move decompress_rest_request_body method into function

3b42172

Co-authored-by: Felix Arntz <[email protected]>

Indicate compression factor in console log

5149289

Perform compression of URL metrics when URL metrics are modified

64c677b

b1ink0 added [Type] Enhancement A suggestion for improvement of an existing feature [Plugin] Optimization Detective Issues for the Optimization Detective plugin labels Mar 27, 2025

Merge branch 'trunk' into add/reintroduce-url-metrics-gzip-compression

cb40cab

westonruter reviewed Mar 27, 2025

View reviewed changes

b1ink0 and others added 3 commits March 28, 2025 16:12

Fix debouncing logic for URL metric compression

12af266

Co-authored-by: Weston Ruter <[email protected]>

Cancel idle callbacks in URL metric compression to prevent race condi…

aeda409

…tions Co-authored-by: Weston Ruter <[email protected]>

Warn URL Metric compression is disabled due to use of deprecated `fin…

7e8ec60

…alize` function Co-authored-by: Weston Ruter <[email protected]>

westonruter added this to the optimization-detective 1.0.0-beta4 milestone Mar 31, 2025

westonruter reviewed Mar 31, 2025

View reviewed changes

b1ink0 and others added 2 commits March 31, 2025 09:23

Refactor JavaScript type annotations to align them with phpdoc

1d78aaf

Co-authored-by: Weston Ruter <[email protected]>

Add error handling for URL metric compression

6c8c257

Co-authored-by: Weston Ruter <[email protected]>

westonruter reviewed Mar 31, 2025

View reviewed changes

Define compression debounce wait duration as a constant

dc4d5e9

Co-authored-by: Weston Ruter <[email protected]>

b1ink0 marked this pull request as ready for review March 31, 2025 06:39

b1ink0 requested a review from felixarntz as a code owner March 31, 2025 06:39

b1ink0 requested a review from westonruter April 2, 2025 10:02

This comment was marked as spam.

Sign in to view

Improve eslintignore pattern for minified JS files

9355b42

westonruter added 10 commits April 3, 2025 11:13

Add jsdoc for more module symbols

51c3dd0

Use extension module URL in logging when no name is available

3f6411a

Move finalize deprecation warning to run earlier to improve discovery

2d37234

Only warn about finalize preventing compression if it was enabled to …

2f5ab44

…begin with

Derive extension name from script module URL

67a203a

Log the extension module script URL with warnings and errors

cb863ea

Move docs to hooks.md

f41bf04

Merge branch 'trunk' of https://github.com/WordPress/performance into…

4eb5a6a

… add/reintroduce-url-metrics-gzip-compression

Use const and fix typo

15d58c9

Supply missing base URL

6aa261c

westonruter approved these changes Apr 3, 2025

View reviewed changes

westonruter merged commit ac1a6ad into WordPress:trunk Apr 3, 2025
19 checks passed

westonruter mentioned this pull request Apr 3, 2025

Ensure URL Metric is initially constructed with all elements prior to initializing extensions #1968

Merged

westonruter reviewed Apr 3, 2025

View reviewed changes

westonruter mentioned this pull request Apr 3, 2025

Explore use of the fetchLater API should be used instead of using keepalive fetch at pagehide when available #1970

Open

	* @type {ReturnType<typeof setTimeout> \| null}
	* @type {number\|null}

Add gzip compression for URL metrics using Compression Streams API #1959

Add gzip compression for URL metrics using Compression Streams API #1959

Uh oh!

Conversation

b1ink0 commented Mar 27, 2025

Summary

Relevant technical choices

Uh oh!

codecov bot commented Mar 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

b1ink0 commented Mar 27, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

westonruter commented Mar 27, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Mar 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment was marked as spam.

westonruter left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

westonruter Apr 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

codecov bot commented Mar 27, 2025 •

edited

Loading

github-actions bot commented Mar 31, 2025 •

edited

Loading

westonruter Apr 3, 2025 •

edited

Loading