This is an important bit in the #Cloudflare post (emphasis mine):

"Our data analysis focuses on traffic from Internet properties on Cloudflare’s free plan, which *includes leaked credentials detection as a built-in feature.*"

@buherator It's a valid point but in the description of the service, it does not specify anywhere that they are using hashed credentials for their lookups. There is no reason to assume that they didn't collect and analyze plaintext creds on random workstations. Even if it's just the free plan, it's still a very bad look.

@cR0w I mean users seem to have explicitly asked CF to look at the credentials passing through them. I don't get how workstations come to the picture, please clarify!

@buherator I meant whoever was doing this analysis had to have that data, and since there is no explanation to how the passwords themselves were analyzed, we can't assume that they were not plaintext and they stayed in any reasonably secure environment.

@cR0w Mishandling the data is surely a concern, but I don't think this particular case is an indicator of such misuse:
HIBP API is anonymized in the first place. They must already have an "even more" anonymized yes/no signal from their detection service (whether it's using the anon API or a full HIBP copy), and at CF's scale I don't think anyone wants to receive all the non-anonymized request fragments for perf/bandwidth reasons alone.

Sure there may be an evil team at CF who secretly look at creds, but this stat is not an evidence of that.

@buherator I wasn't necessarily implying an evil team, more of one with different levels of due care than we would expect from an org like that. I would have expected that they would be using the looking system in place for HIBP, but they never said anything resembling reassurance that they did anything legitimately.

Even simply saying that they used the hashed passwords for their lookups would have put me much more at ease with this, but I don't trust Cloudflare. A large part of their operations knowingly protects evil shit. And then there's the "anything for a few more dollars" approach by big tech in general. How would that be such an accidental oversight by a security team posting a security blog post for a security audience? Smells very fucking fishy to me.

@cR0w I agree the post is far from optimal, but try to look at it this way: in a system as big as CF's you rarely see individual request data, but you can correlate HIBP alerts with status codes and write a blog post with big %s about password stuffing. You don't write about anonymization because you never saw anything that'd need anonymizing so there's nothing you did that's worth writing about. Again, I agree this should've been flagged by PR, but we've seen bigger blunders from that department...

#HanlonsRazor

@buherator I'm with you that they likely didn't have it, but I'm already biased with hate and even if the project itself was done legitimately, the post is the face of the org so I cannot assume that the absence of evidence of fucking up is not evidence of absence of fucking up. An org with their history needs to be up front and clear that they are about their due diligence and due care when handling sensitive information.

@cR0w @buherator The simple "no need to assume malice" answer is that they didn't mention whether they were using hashes because they assumed their audience assumes they aren't idiots.

But your meta-point, that we don't know for sure unless there's some independent third-party audit of their internal practices, stands. This post suggests they have received third-party audits, but I'm not nearly familiar enough with the standards to tell at-a-glance whether they should give some assurance that this was done by, say, putting together an audit script to run in their data warehouse without any human having the ability to touch the raw passwords... or slurping a bunch of passwords to someone's workstation and running haveibeenpwned database dumps across all of them.

@buherator @cR0w Whether it should have been flagged by PR is, I think, a question of target audience. I can see a Schrodinger's PR Agent here that either asks them to add something about hashing to assuage the tech-savvy crowd... Or asked them to remove some detail about hashing because the target audience isn't the tech-savvy crowd, it's the kind of people who reuse passwords across services, and they don't even know what a "hash" is.

@mark @buherator I'm not assuming malice. I'm simply not assuming legitimacy. Whether they did a dumb on purpose or not isn't really what my risk model is about. The impact is what matters. But assuming that they didn't mention due care because "they assumed their audience assumes they aren't idiots" is quite an assumption in itself.

@cR0w

I'm not assuming malice. I'm simply not assuming legitimacy.

Ah, I see where you're coming from. I missed the distinction because since they're a network connectivity and security provider, I didn't see daylight between malice and illegitimate (equivalent outcomes).

But assuming that they didn't mention due care because "they assumed their audience assumes they aren't idiots" is quite an assumption in itself.

An alternative hypothesis is the target audience for this post isn't the people who care whether they did the analysis using hashed passwords in a humans-not-accessing-directly data lake or plaintext on someone's desktop; it's the kind of people who reuse passwords and don't even know what a "data lake" or a "hash" are.

@mark Plausible deniability requires benefit of the doubt, which Cloudflare pissed away long ago.

@cR0w @mark As I see what needs clarification here is the security/privacy guarantees of the HIBP system (that's been around for some time), as that is the one accessing sensitive data. In a good architecture it should be impossible for that data to leak during/after real-time analysis, making the dispute about this particular statistic mute.

And again, let's not forget that participating origins agreed to this type of data use (hell, they may even need to configure what fields to snoop on!) - I wonder if their end-user privacy policies include this detail...

@cR0w @mark In fact, it was CF that contributed k-anonymity to HIBP:

https://blog.cloudflare.com/validating-leaked-passwords-with-k-anonymity/

@buherator @mark That seems like quite the miss by the bloggers then. A simple reference to their own blog post would have avoided all of this to a certain extent.

That said, while I reasonably trust the HIBP API for what it is, it does not mean that the user does not have a trove of plaintext passwords that are then hashed and submitted. That's where my concern was.

@cR0w @mark Yes, a MitM-as-a-Service provider *may* see and misuse your passwords.

Does this particular stat make any difference to that equation? No.

"They have more compromising data than they could possibly exploit" doesn't really reassure me. You know it's the malicious who want you to attribute everything they do to stupidity, right?

CC: @cR0w@infosec.exchange

@buherator @mark Agreed. The post came across as "look what we're doing with your passwords without privacy concerns" though, and given the rest of the shady stuff they do, makes the risk less theoretical and more likely to be occurring, regardless of the reality. Optics matter and this is exactly the kind of article I can see going around LinkedIn and having an executive come to the ICs of any org with a giant "WTF is Cloudflare doing?"

@cy @buherator I agree with your point but Buherator and I actually had what I think was a good discussion further down the thread. Or down a thread. I tend to butcher threads on here. 😆

@cy @cR0w If you read carefully you'll see that I applied Hanlon's Razor to the blog post, not the operational practices. On that part my argument is that they'd need to go far out of their way to do evil, which doesn't mean they don't do it, but I'm pretty sure they won't do it for a security-awareness blog post.

Oh yeah the blog post was definitely stupidity. Still not very reassuring to see!

CC: @cR0w@infosec.exchange

@cR0w I dunno, 24.03 million websites seem to disagree. 🤷

@mark I live in America. You cannot convince me that just because millions of people disagree with me that maybe I'm the one who's wrong. I'm full Seymour Skinner here and I own it.

@cR0w I'll drink to that.

Personally, what I'd love is a better option. I'm not thrilled about using a reverse proxy through someone else's permanent anchor to "the real internet" to host services from machines in my house. But I've tried alternative solutions for anything even vaguely approximating a secure connection (without risking, in general, exposing my intranet to attackers), and... It's kinda a hellscape out there?

Cloudflare is providing a hell of a convenient service that it'd be nice if someone else could provide without doing what Cloudflare does. I fear the architecture of the Internet herself somewhat precludes it.

@mark Sounds like you need Wireguard and a VPS. That's what I do and it works great. I'll add it to my growing backlog of things I need to do a blog post about. It's a really simple setup.

There were plenty of better options a decade or so ago. We need antitrust legislation at this point I'm afraid.

Though it begs the question, why use Cloudflare at all? Are attackers really that big of a problem, that you'd want the equivalent of mafia insurance, giving up 100% of your intranet's security, in exchange for a promise of not being attacked?

I mean it is a problem for some. My "intranet" has never had problems with attackers though. Would a VPN do what you want? Still lots of those hanging around.

CC: @cR0w@infosec.exchange

@cy @mark "We paid Cloudflare for security service."

"Cool, Compliance is happy."

@cR0w But then I'd have to pay more than $0, right?

@mark My lower usage ones are $3.50 / month. To me, it's worth it but you are correct in that it is greater than $0 and that comes with other costs besides the money itself.

@cR0w Actually, $3.50 / month is low enough to make me consider it. I was estimating from what it would take to spin up a proxy in a cloud provider like AWS or Google and that was coming out to something closer to $20 (with a risk of being real expensive if I eff up the config and somehow slam the instance to 100% CPU for all the time it's up).

@mark You can get a Lightsail ( AWS lite ) instance for $3.50 / mo. That's IPv6 only though and another $ for IPv4 address. Pretty much all the big providers have a $5 / month option though and if it's for a server rather than client Internet access, I have had good luck over the years with Linode.

@cy @cR0w Me personally, I'm not using it because I'm concerned about attackers. I'm using it because it lets me put a pinhole into my outward-facing firewall that accepts only one kind of traffic at the application level which, by default, doesn't even expose my IP address to the outside world (traceroutes stop at Cloudflare's network), and if my service provider changes my IP assignment, the reverse IP will self-heal.

you'd want the equivalent of mafia insurance, giving up 100% of your intranet's security

Here's the thing about big companies: they're also big targets for regulation. The larger the company, the more I can trust that if they do something truly disastrous, the government will care enough to come down on them like bricks (and if the government doesn't, shareholders or customers fleeing will make it financially painful enough that they have a reason to not drop the ball).

There should be a someone's law about this. It's like Too Big To Fail but for infra providers.

Trouble is the bigger they are, the relatively smaller disasters are. Sufficiently big, your total ruin would just be a blip on the radar for them. No one could come down on them because it'd destroy the entire Internet. Anyway, our threat models are different. I'm more worried about losing opportunity in life to gigantic and rich companies, than whether some evil hacker gets my IP. You could very well feel differently.

a pinhole into my outward-facing firewall that accepts only one kind of traffic at the application level which, by default, doesn't even expose my IP address

Sounds like a VPS or VPN to me. Except those won't make you break your encryption for them.

CC: @cR0w@infosec.exchange

@buherator do we take it that paid plans do *not* include this "feature" then?

@infosecdj no idea, but I'm sure it's documented by cf somewhere...