CloudFlare blocking requests

[jcconnell]

It appears CloudFlare is blocking requests now. The body of the request returns error_code: 1020 while the status code is a 403. I think it's something to do with CloudFlare's bot protection.

[mhelmetag]

Yeah... looks like the CloudFlare is blocking requests that don't specify a Host header in the request (I'm not supporting that in my package at the moment...). I'll see what I can come up with! Surfline is cracking down.

I played around in Postman and was able to still have requests go through so not all is lost.

Thanks for the heads up! I've already had to take some services down because of some other recent CloudFlare protections.

[jcconnell]

I think it's fingerprinting the handshake. Curl works if you add a User-Agent header but fails without it. Curl seems to upgrade its connection to TLS 1.2 and HTTP2 along the way but I can't replicate the behavior with changes to the Go net/http client (yet).

I've tried the following without success so far:

• Tried using HTTP/2 transport in the HTTP client
• Tried adding the same headers the browser uses

[mhelmetag]

Yeah... I realized I was looking at another endpoint. I think some endpoints are more locked down than others. This might be a tougher one. Dang.

[jcconnell]

Not sure if this differs from yesterday, but the GetTides() query appears to get a successful response but fails decoding. The decoded response looks like: Tides: {{{F FT FT FT KTS } -10 {0 0}} {[]}}

[mhelmetag]

Can you try again? Seems to be working again for me... I also see a difference in how the Surfline web app is making request now. Hopefully that wasn't a taste of what's to come 🤞 but... I have a feeling it might be

[jcconnell]

Nice, working again! Thanks for the follow up. Also hoping it continues to work!

[TrevHeath]

Seems to still be an issue on the server calls where the browser request headers are not available. Did you guys find a solution?

[mhelmetag]

Ah, yeah... seems like we're back to being blocked. I'll make a patch that allows the inclusion of headers into the client.

[mhelmetag]

Hmm, this is proving hard to test since it only seems to happen some days...

[mhelmetag]

Also, sometimes servers can be completely blocklisted. I've had one that I had to tear down and start fresh. Things started working again after that. Also, it does seem like https://github.com/swrobel/meta-surf-forecast has been having long term luck. I wonder if it's because he always requests with an access token. I'm sure that's where Surfline will end up eventually but they aren't there yet. That said.... I don't really believe the headers do much.

[mhelmetag]

Alrighty, headers definitely do a lot. Also using cookies (since cloudflare likes those) and being logged in.

[mhelmetag]

#14

[mhelmetag]

I haven't had any recent troubles with this... I'll keep that issue for adding headers open as I think that's a good idea anyways.