We've been seeing the upload-to-s3 step frequently fail on EWS archive builders without any logging, causing them to fallback to the "upload to master first" flow, which is frequently taking over an hour to complete. We attempted to add logging in https://commits.webkit.org/266556@main, but we're still not seeing any indication of why the step is failing. Some examples from today: https://ews-build.webkit.org/#/builders/14/builds/14897 https://ews-build.webkit.org/#/builders/16/builds/14637
<rdar://problem/113416993>
First of all, uploads taking 1+ hour indicate that the buildbot master was overloaded. Secondly, this particular upload method is used only when 'upload-file-to-s3' step fails (i.e.: bots weren't able to directly upload to s3, so as a fallback they upload to master). Logs were added in https://commits.webkit.org/266722@main (Bug 259929). Logs indicate that the issue is in generate-s3-url step. It generates the logs, but for some weird reason it fails to parse the logs (in order to parse the generated url). For e.g. in https://ews-build.webkit.org/#/builders/14/builds/15521 generate-s3-url step does have the url, but its status is: "Failed to generate S3 URL". When this issue happens 'log_text' variable is empty in GenerateS3URL::run(). Not sure why. It might be some buildbot issue when the server is overloaded (based on the fact that this seems to happen when the server is overloaded and upload to master takes hours instead of minutes).