Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

503 on publishing a crate #10098

Closed
alexjg opened this issue Nov 28, 2024 · 13 comments · Fixed by #10102
Closed

503 on publishing a crate #10098

alexjg opened this issue Nov 28, 2024 · 13 comments · Fixed by #10102

Comments

@alexjg
Copy link

alexjg commented Nov 28, 2024 •
edited
Loading

Current Behavior

I'm running cargo publish from within the crate I'm trying to publish (automerge_repo_rs) and I consistently get a 503 back from crates.io. I've tried refreshing my API token just in case that's the issue to no avail.

Here's an example response:

error: failed to publish to registry at https://crates.io

Caused by:
  failed to get a 200 OK response, got 503
  headers:
  	HTTP/2 503
  	content-type: text/html; charset=utf-8
  	content-length: 506
  	cache-control: no-cache, no-store
  	server: Cowboy
  	date: Thu, 28 Nov 2024 14:23:30 GMT
  	report-to: {"group":"heroku-nel","max_age":3600,"endpoints":[{"url":"https://nel.heroku.com/reports?ts=1732803811&sid=af571f24-03ee-46d1-9f90-ab9030c2c74c&s=9dHZ0W9GZnop99SEhzT0Dw2wrnjV3uEV9NgaUgnJvIM%3D"}]}
  	reporting-endpoints: heroku-nel=https://nel.heroku.com/reports?ts=1732803811&sid=af571f24-03ee-46d1-9f90-ab9030c2c74c&s=9dHZ0W9GZnop99SEhzT0Dw2wrnjV3uEV9NgaUgnJvIM%3D
  	nel: {"report_to":"heroku-nel","max_age":3600,"success_fraction":0.005,"failure_fraction":0.05,"response_headers":["Via"]}
  	x-cache: Error from cloudfront
  	via: 1.1 78ddfb54ad788c32ecf2935db37685b8.cloudfront.net (CloudFront)
  	x-amz-cf-pop: LHR62-C4
  	x-amz-cf-id: bbu6Bb5oolK_Vxd0YxQLuADCXI8wNm22AIElvoKaUFga_pntIclrdQ==
  	strict-transport-security: max-age=31536000; includeSubDomains
  	
  body:
  <!DOCTYPE html>
  	<html>
  	 <head>
  		<meta name="viewport" content="width=device-width, initial-scale=1">
  		<meta charset="utf-8">
  		<title>Application Error</title>
  		<style media="screen">
  		 html,body,iframe {
  			margin: 0;
  			padding: 0;
  		 }
  		 html,body {
  			height: 100%;
  			overflow: hidden;
  		 }
  		 iframe {
  			width: 100%;
  			height: 100%;
  			border: 0;
  		 }
  		</style>
  	 </head>
  	 <body>
  		<iframe src="//www.herokucdn.com/error-pages/application-error.html"></iframe>
  	 </body>
  	</html>

Expected Behavior

I expect cargo publish to successfully publish the crate to crates.io.

Steps To Reproduce

Obviously this will only work for me, but the steps I take are:

  1. Clone https://github.com/automerge/automerge-repo-rs
  2. git checkout v0.2.0
  3. cargo publish

Environment

  • OS: Arch Linux
  • rustup show

    Default host: x86_64-unknown-linux-gnu
    rustup home:  /home/alex/.rustup
    
    installed toolchains
    --------------------
    
    stable-x86_64-unknown-linux-gnu (default)
    nightly-x86_64-unknown-linux-gnu
    1.78-x86_64-unknown-linux-gnu
    1.81-x86_64-unknown-linux-gnu
    
    installed targets for active toolchain
    --------------------------------------
    
    wasm32-unknown-unknown
    x86_64-unknown-linux-gnu
    
    active toolchain
    ----------------
    
    stable-x86_64-unknown-linux-gnu (default)
    rustc 1.82.0 (f6e511eec 2024-10-15)

Anything else?

No response
@alexjg alexjg changed the title 502 on publishing a crate 503 on publishing a crate Nov 28, 2024
@Turbo87
Copy link
Member

Turbo87 commented Nov 28, 2024

hmmmmmmm...... interesting.....

according to our server logs we replied with "413 Payload Too Large", but apparently Heroku turned that into a "503 Service Unavailable"? 🤔

Image

to fix the immediate issue: run cargo package and make sure that the generated crate file has less than 10 MB. if it's above that you might want to check what you're bundling, and exclude unneeded large files from the packaging process.

in the meantime I'll try to find out why this is turning into a 503 error

@liulifox233
Copy link

I encountered the same issue. After running cargo package, I found that the generated .crate file was only 23KB. However, I am still unable to upload it.

du -sh *
831M	ramdl-1.0.0
 24K	ramdl-1.0.0.crate

Here is my repository: https://github.com/liulifox233/ramdl

@Turbo87
Copy link
Member

Turbo87 commented Nov 28, 2024

yeah, I'm currently checking this on our staging environment, and it seems that currently all error responses from our API are turned into 500 or 503 errors for some reason. in other words: this is not just about the upload size, though in @alexjg's case it was.

@alexjg
Copy link
Author

alexjg commented Nov 28, 2024

Yeah in my case I had accidentally included a bunch of build output from some tests in the crate, fixing that has allowed me to publish the crate.

@Turbo87
Copy link
Member

Turbo87 commented Nov 28, 2024

I'm expecting #10069 to be the cause of these symptoms, but it seems like the root cause is a limitation in CloudFront and/or Heroku.

On our staging environment I get a "500 Internal Server Error" response from CloudFront instead of the 503 above. The difference there being that the staging environment uses the "Heroku Router 2.0" already, while the production environment does not. When I disable that "Heroku Router 2.0" feature I get the same 503 Heroku error as above. In other words: it looks like "Heroku Router 2.0" fixed the issue on the Heroku side, but there still remains a problem on the CloudFront side.

@Turbo87 Turbo87 mentioned this issue Nov 28, 2024
Merged
@Turbo87
Copy link
Member

Turbo87 commented Nov 28, 2024

I'm deploying the workaround to staging right now and will enable it on production too. This should be "fixed" within the next hour. Sorry for the inconvenience!

@alexjg
Copy link
Author

alexjg commented Nov 28, 2024

Thanks for the quick response!

@Turbo87
Copy link
Member

Turbo87 commented Nov 28, 2024

The workaround is now deployed to production too

@eth3lbert
Copy link
Contributor

Is it possible that this could be an issue related to the axum side, similar to this tokio-rs/axum#2850?

@Turbo87
Copy link
Member

Turbo87 commented Nov 29, 2024

yeah, aside from this using multipart uploads this sounds very similar. the linked stackoverflow issue makes it sound like this is not an axum-specific issue though. the open browser bug reports indicate that this is easier to support with Http 2, which might explain why upgrading the heroku router fixed it on that side. unfortunately that still leaves us with cloudfront seemingly not supporting this properly.

@eth3lbert
Copy link
Contributor

I was wondering if read_to_end() before returning the error while streaming would help 🤔

@Turbo87
Copy link
Member

Turbo87 commented Nov 29, 2024

that might be an option but we would have to do so in every single error case that we return, potentially including the ones from middlewares. and I think read_to_end() requires a buffer, but we should probably just discard the data without buffering it.

I plan on adjusting our smoke test today to check for auth failure in the upload path, which should have caught the problem early. I'll also try out whether cargo can handle early responses by going around cloudfront. if cargo can't handle it then we're out of luck anyway, but from what I've read on the linked issue curl handles this correctly and AFAIU cargo uses curl by default.

@Turbo87 Turbo87 mentioned this issue Nov 29, 2024
Merged
@Turbo87
Copy link
Member

Turbo87 commented Nov 29, 2024

I'll also try out whether cargo can handle early responses by going around cloudfront.

I've tested this down to cargo 1.49, since earlier versions showed "target 'aarch64-apple-darwin' not found in channel" errors when I tried to install them on my system. On all versions I've tested it worked fine once Cloudfront was out of the way...

@Turbo87 Turbo87 mentioned this issue Nov 29, 2024
Merged
@Turbo87 Turbo87 mentioned this issue Dec 9, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Revert "Merge pull request #10069 from Turbo87/publish-stream" Turbo87/crates.io
4 participants
@Turbo87 @alexjg @eth3lbert @liulifox233