Bug 259842

Summary:	[EWS] upload-file-to-s3 step frequently failing, uploads to master taking > 1 hour
Product:	WebKit	Reporter:	Ryan Haddad <ryanhaddad>
Component:	Tools / Tests	Assignee:	Aakash Jain <aakash_jain>
Status:	ASSIGNED ---
Severity:	Normal	CC:	aakash_jain, ap, jbedard, ryanhaddad, webkit-bot-watchers-bugzilla, webkit-bug-importer
Priority:	P2	Keywords:	InRadar
Version:	WebKit Nightly Build
Hardware:	Unspecified
OS:	Unspecified
See Also:	https://bugs.webkit.org/show_bug.cgi?id=259929

Description Ryan Haddad 2023-08-04 15:43:33 PDT

We've been seeing the upload-to-s3 step frequently fail on EWS archive builders without any logging, causing them to fallback to the "upload to master first" flow, which is frequently taking over an hour to complete.

We attempted to add logging in https://commits.webkit.org/266556@main, but we're still not seeing any indication of why the step is failing.

Some examples from today:
https://ews-build.webkit.org/#/builders/14/builds/14897
https://ews-build.webkit.org/#/builders/16/builds/14637

Comment 1 Radar WebKit Bug Importer 2023-08-04 15:44:00 PDT

<rdar://problem/113416993>

Comment 2 Aakash Jain 2023-08-23 09:26:03 PDT

First of all, uploads taking 1+ hour indicate that the buildbot master was overloaded. Secondly, this particular upload method is used only when 'upload-file-to-s3' step fails (i.e.: bots weren't able to directly upload to s3, so as a fallback they upload to master).

Logs were added in https://commits.webkit.org/266722@main (Bug 259929). Logs indicate that the issue is in generate-s3-url step. It generates the logs, but for some weird reason it fails to parse the logs (in order to parse the generated url). For e.g. in https://ews-build.webkit.org/#/builders/14/builds/15521 generate-s3-url step does have the url, but its status is: "Failed to generate S3 URL".


When this issue happens 'log_text' variable is empty in GenerateS3URL::run(). Not sure why. It might be some buildbot issue when the server is overloaded (based on the fact that this seems to happen when the server is overloaded and upload to master takes hours instead of minutes).