📣THREAD: It’s surprising to me that so many people were surprised to learn that Signal runs partly on AWS (something we can do because we use encryption to make sure no one but you–not AWS, not Signal, not anyone–can access your comms).

It’s also concerning. 1/

Concerning, bc it indicates that the extent of the concentration of power in the hands of a few hyperscalers is way less widely understood than I’d assumed. Which bodes poorly for our ability to craft reality-based strategies capable of contesting this concentration & solving the real problem. 2/

The question isn’t "why does Signal use AWS?" It’s to look at the infrastructural requirements of any global, real-time, mass comms platform and ask how it is that we got to a place where there’s no realistic alternative to AWS and the other hyperscalers. 3/

Running a low-latency platform for instant comms capable of carrying millions of concurrent audio/video calls requires a pre-built, planet-spanning network of compute, storage and edge presence that requires constant maintenance, significant electricity and persistent attention and monitoring. 4/

Instant messaging demands near-zero latency. Voice and video in particular require complex global signaling & regional relays to manage jitter and packet loss. These are things that AWS, Azure, and GCP provide at global scale that, practically speaking, others (in the western context) don’t. 5/

This isn't ‘'renting a server.' It's leasing access to a whole sprawling, capital-intensive, technically-capable system that must be just as available in Cairo as in Capetown, just as functional in Bangkok as Berlin. Particularly given the high stakes use cases of many who rely on Signal. 6/

Such infrastructure costs billions and billions of dollars to provision and maintain, and it’s highly depreciable. In the case of the hyperscalers, the staggering cost is cross-subsidized by other businesses–themselves also massive platforms with significant lockin. 7/

Meaning that infrastructure like AWS is not something that Signal, or almost anyone else, could afford to just “spin up.” Which is why nearly everyone that manages a real-time service–from Signal, to X, to Palantir, to Mastodon–rely at least in part on services provisioned by these companies. 8/

But even if Signal had the billions needed to recreate AWS, it’s not just about money. The talent to run these systems is rare & concentrated. The expertise, the tooling, the playbooks, the very language of modern SRE came out of these hyperscalers, and is now synonymous with 'the cloud.' 9/

o, yes, Signal runs on AWS. It also runs on your phone, which runs on iOS (Apple) or Android (Google). And on Dekstop, via Windows (Microsoft). Each of these presents similar dependencies on large entrenched tech companies, and concomitant barriers and risks. 10/

In short, the problem here is not that Signal ‘chose’ to run on AWS. The problem is the concentration of power in the infrastructure space that means there isn’t really another choice: the entire stack, practically speaking, is owned by 3-4 players. 11/

So, Signal does what we can to provide a service w integrity in the concentrated ecosystem we're working in. We protect your comms w end-to-end encryption, so that we can use AWS and others as a highway across which to send Signal data in ways that don’t let AWS, or anyone else, gain access. 12/

To conclude: my silver lining hope is that AWS going down can be a learning moment, in which the risks of concentrating the nervous system of our world in the hands of a few players become very clear. And that this can help us craft ways of undoing this concentration and creating real choice ❤️ 13/

@Mer__edith

The tor network has had 100% uptime. 100%

@yawnbox I don't think you have a clear understanding of what you're talking about, and it might be fun for you to look a bit more deeply into how TOR works and its dependencies.

@Mer__edith Agree - if you want to run your service centralized. Neither my Mastodon nor my Matrix-server need anything but my own self-hosting. Of course they won't handle billions of concurrent customers - but a few tens of thousands similar to mine will. Together.

I simply don't think Signal being centralized is a good thing. It's your choice, but alternatives do exist and those do not need hyperscalers.

@troed I don't think you have a clear understanding of this space, but I hope you have a good time digging in and learning more.

@Mer__edith interesting engagement levels across different sites.

@Mer__edith great explanation and thank you for that.

Is it possible to build a future promoting other competitors to AWS gathering partners (other executive leadership with different companies) who rely heavily on these services to help level the playing field as well as give applications like Signal multiple options to run their infra? It would be complex, but it already is anyway. I've learned over the years to be as redundant as possible in every aspect of your infrastructure which includes things like cloud providers.

I have ran redundant workloads in AWS, GCP, Azure and tried to tie that in with smaller orgs like Linode, OVH and have had success. I understand that Signal is far more complex and your service needs to be real time whereas in oil and gas(the industry I was in mainly during my career) not everything needed to be as real time as your application.

We need someone to kickstart us into balancing out the playing field to give other, smaller competitors a chance at the same time helping to make these applications more redundant in the event of a disaster like we just had. Costs for such a design would be high but overtime that could balance out as we get more competition flowing.

@Mer__edith Is there any chance of Signal adding its own keyboard in the future? The input system seems like a pretty important privacy concern at the moment.

@Mer__edith How about multi-cloud? Like if AWS fails direct traffic towards GCP (but who directs the traffic ik...)? Do you do this? Is it possible in your case?

@Mer__edith It's possible to manage a real-time service without these companies, but it's challenging. I understand that it would hard for Signal to build and maintain its own infrastructure, but the questions is, if Signal team is considering what to do in the future to make impact of future outages (which may occur again) to have lesser impact on people who use it.

@Mer__edith Thanks for your condescending reply. I used to manage global SaaS within fintech with nodes in GCP, AWS and Azure and on multiple different continents.

@Mer__edith Hello Meredith, all fair points of course, and I believe that many understood what happened. Isn't there a way to distribute the load in a fashion that it would not fail when US-EAST-1 at AWS is down ? (either with AWS itself or with other providers to balance the risks)

@ericfreyss @Mer__edith
presumably that's on AWS to fix. the AWS deployment of Signal uses DynamoDB, which in theory is multi-region. however that service has a chokepoint in US-East-1 that is part of the AWS design and not something anyone other than them can fix - you can't choose to have the DNS for it be deployed somewhere else when you deploy to DynamoDB. it's AWS's task to make DynamoDB actually as multi-region as they advertise it being.

@Yuvalne @ericfreyss @Mer__edith

This was the comment I was hoping to find!

The issue seems to be how AWS itself is internally architcted for resilience, or in the case of the last outage contains points of failure.

This isn't surprising but it does present challenges. I doubt that AWS will fail the same way again but next time it will be some different unanticipated failure mode.

Hopefully AWS will now be looking at this and hopefully coming up with a solution 🙂🤷‍♂️

@troed @Mer__edith I might be missing your point but there does need to be A choice that doesn't require self-hosting; because I promise you my parents are not about to learn to run a Signal-like service for themselves, and I don't want to have to do it for them.

Otherwise services like WhatsApp exist, and they just work, so they'll just use that. Which means I have to use it too. The value of a messaging app is (IMO) ENTIRELY derived from who you know on it so it has to be easy to adopt.

@_calmdowndear

There's something inbetween "everyone selfhosts" and "everyone uses a single instance". Since you're posting on Mastodon, I do think you know what it is ;)

@Mer__edith

@buherator We use multiple cloud providers, yes.

@Mer__edith Then did your failover fail? Would you be able to handle a similar event in the future based on lessons learned or are there technical constraints to survive DynamoDB going dark?

@troed yes but I'm posting from a "centralised" Mastodon instance (by which I mean it's outside of my control and there is a network effect beyond just me when this instance experiences issues).

I'm on Mastodon because the people I'm interested in following are here. But I have less than zero interest in self-hosting my own server.

This is only a partial solution IMO, and only works as well as it does because Mastodon is not required to be "real time" like a voice call.

@_calmdowndear So, your parents don't need to self-host yet they can be on a network that doesn't require everyone to be on the _same_ centralized instance.

My other example was Matrix. It's like Signal, but decentralized like Mastodon. It fully supports voice and video calls.

@troed@swecyb.com Those explanations about how something like Signal is not possible or pragmatic without AWS or the other big players felt like gaslighting and the result is that I no longer trust Signal. Going as far as suggesting "mastodon" (meaning actually the fediverse) also requires AWS is disingenuous at best and malicious at worst, specially considering the CEO was using the fediverse to communicate about the Signal outage as it was happening. Arguing that you need to rely on Android (Google), iOS (Apple) or Windows (MS) to run the client is straight lying as one can use the desktop version to run on Linux and requiring a mobile app to sign up is a choice Signal did, a problem of their own making. Yes I know there is a lot of nuance but the end result is the same: trust has been lost.

@troed so "Mastodon is down" is a thing that could still happen if the right piece of infrastructure dies. Because that's what happens from the user perspective - yes, the network may be up but if "your" instance is dead and you can't interact with it, then who cares?

I can't comment on Matrix too much as I hardly ever use it (unless required by some software using it for support). But what I have seen of it is a very frustrating user experience.

@_calmdowndear Absolutely - just like your local ISP can go down and those affected say "the Internet is down".

The rest of the world still works fine though.

@Mer__edith

nearly everyone that manages a real-time service–from Signal, to X, to Palantir, to Mastodon–rely at least in part on services provisioned by these companies

Mastodon doesn't, though?

There certainly will be servers hosted on AWS but when AWS went down, most Mastodon instances stayed up, and people were cracking jokes at more centralized platforms.

@datum Mastodon is distributed at the level of the protocol, not infrastructure. Sure, some people use a server in their closet, but most license hyperscaler infra to host their mastodon instance.

Meta note, we seem to be dealing with a confusion in what the term "distributed" means in this context.

@Mer__edith Thanks, your explanation is helpful. Does it suggest that text-messaging should perhaps be the focus? I'm amazed that video/audio is possible, but it incurs much more massive requirements - it seems that the video/audio service is constraining the possible infra choices for the text-message service.

@danstowell No, because no modern and useful comms platform provides only text messaging. And if your platform doesn't provide normative features and functionality, people won't use it. And if they don't use it--even if you do--you can't use it either. The network effect rules communications effectiveness.

@Mer__edith The reason AWS is affordable is because they are *still* subsidizing your invoices by 80%. When doing price comparisons assume that one day AWS will start charging 5x more once they have succeeded in killing the rest of the competition. All of them will. Im certain if AWS was charging what it costs to run that service that colo or any other option would be just as viable from a price perspective. They absolutely cannot compete on anything other than price. They are really bad at it

@buherator not only does Signal already do that as @Mer__edith mentioned, but routing to GCP is exactly what Signal did during the latest outage, hence why for many users it recovered before AWS did.

@datum

It's easy to gloat about AWS but if Hetzner goes down it takes half the Fediverse with it. Same problem, different vendor.

@Mer__edith

@Mer__edith I don't understand one thing. You mention Mastodon and to me that seems a bit inaccurate? I'm not disagreeing with you in how the prevalence of big scale cloud providers is bad, and how Signal as a centralised platform basically doesn't have other options. But I'm just wondering, isn't decentralisation one way to solve this?

The whole thread is about how AWS and other big scale providers created the international infrastructure that is needed for Signal to run smoothly and that infrastructure would be impossible for Signal to set up independently. But Mastodon (and Matrix) is decentralised, so yes, if there was one big Mastodon server it would need these resources, but isn't it one perk of decentralisation that because there are so many independent people doing it, creating that international infrastructure becomes possible?

I've been using Signal for almost 10 years and Mastodon/fediverse for around 3 including hosting my own instances (both self-hosting and using small-medium scale server providers) and they have been very comparable in speed and reliability.

@marta Decentralization at the level of protocol--in Mastodon's case via ActivityPub--does not mean decentralization at the level of infrastructural dependence.

@Mer__edith maybe Signal should reevaluate its standpoint against federation then? This would enable different entities to run different parts of the network in different regions and will remove the need for a globe spanning compute platform...

@Yuvalne @buherator I appreciate you @Yuvalne

@Mer__edith @troed This is a really shitty reply, especially when multiple experienced people are expressing the same opinion.

@JadedBlueEyes @troed I'm sorry if it landed harsh.

First, I don't think not knowing things is inherently bad or shameful--it's where learning starts, etc. Second, there's a misunderstanding here, whoever is expressing it: decentralization at the level of a protocol--ActivityPub or w/e else--is NOT the same thing as decentralization of infra. People running mastodon instances, or Matrix servers, or other fedi systems, are also in most cases leasing infrastructure from hyperscalers to do so.

@Mer__edith

You're claiming something that isn't true (no, most people are not using hyperscalers - and yes - I know many people who run Matrix-instances) - and then respond to people that we don't have a clear understanding of this space.

It didn't "land harsh", btw. It just completely changed my impression of you since you actually seem to believe it.

@JadedBlueEyes

@Mer__edith @buherator thank you. I'm just a Signal nerd who spent like half of the outage being on the forum and I'm in essence just repeating stuff said there

@Mer__edith

With respect Meredith, i’m talking about decentralized protocols and their capability to not depend so heavily on the service providers you’re arguing for. Tor Project has shown how possible it is (i used to work there, and it’s spelled Tor not TOR).

I listened to Moxie’s aversions to decentralization for years. That’s what I keep seeing now, with posts like these. I also understand the value of huge cloud providers, I’ve worked for many companies who use them, and have worked for them, and I understand why you depend on them and how important that is to a high quality service. Thank you for all that you all do.

But what conversations does Signal Foundation actually have on the topics of resiliency through decentralization? How much money could you save by allowing the community to take on aspects of the network? How much resiliency and trust could be gained, without losing performance?

@Yuvalne @Mer__edith Oh I didn't realize that was happening as Signal recovered at about the same pace as AWS did (also status page shows a binary value so it's hard to tell the overall state)! Thanks!

@alessandro

Hetzner isn't a hyperscaler though. Meredith seems to confuse people choosing a VPS instead of hosting at home with using advanced AWS (et.al) services for scaling.

There seems to be a real disconnect in understanding here.

https://en.wikipedia.org/wiki/Hyperscale_computing

@datum @Mer__edith

@buherator @Mer__edith i know users in different regions got service at different times. i remember someone mentioning on the forum that Singapore was one of the hardest hits, taking many hours to come back online. and of course there were the users who weren't affected at all, which to my understanding were mostly in North America.
i got service back in Central Europe relatively early on, when AWS was still coming back online.

@Yuvalne Can you maybe point me to a forum thread with technical details (what did Signal do/see, that sort of thing) about the failover/recovery?

@troed

What's the difference if everybody uses the same few VPS providers? The technical details of a massive outage don't really matter - the problem is that we have very large single points of failure, but the immense technical demands of modern tools mean that there are few alternatives.

@datum @Mer__edith

@buherator
this is the thread i'm taking my info from.
https://community.signalusers.org/t/signal-service-outage-2025-10-20/72317?u=rassilon1963

@alessandro

If you use AWS (et.al) services it's difficult to move your SaaS to another provider. If you're using a VPS it's trivial.

(And we aren't all using Hetzner, or OVH, etc)

@datum
@Mer__edith

@yawnbox @Mer__edith Tor is basically a glorified network protocol (albeit very smart) so having it distributed by design is less of an issue.

I agree that making Signal more robust through decentralisation would be great, but this sort of thing gets more difficult the higher up the stack you go, especially when it wasn't part of the core design principles.

@davep @Mer__edith like count here also may not be accurate unless you view it from mastodon.world

@Yuvalne Thank you!

@yawnbox @Mer__edith I think moxie made a pretty clear point on his view of decentralized vs centralized with his ccc-talk a few years ago.

That aside, Tor may be able to provide in terms of uptime, but latency? If I'd be in signals boots I would not build on a network run by people on all kinds of hosting with no sufficient ability to have some kind of guarantee on the QoS aspect.

@Mer__edith Thank you for the Tread. I did not know. 😃

@Mer__edith
I remember SETI distributed a program that could, if installed on PC's, help them to run calculations to detect alien intelligence signs from radiotelescope observation files.

I wonder why there wouldn't be an option on signal desktop to allow PC's to be able to carry part of the traffic as a node (like Session messenger does).

This could help alleviate the effects of the next AWS outage, wouldn't it?

@Teratogenese @Mer__edith History has an example of Skype (pre microsoft) "super nodes". It was very problematic.

@Mer__edith What was surprising to me was that a problem in one AWS availability zone caused a persistent outage for my Signal connection that lasted at least 3 hours. I expected there would be some sort of automated failover to another zone or cloud provider happening in the background, limiting the downtime to, say, maximum 15 minutes. I hope your internal post-mortem is taking a look at how to be better prepared next time.

@jwildeboer @Mer__edith There’s a thread elsewhere which I have since lost where an SRE pointed out that not all regions are equal: not only is EC1 the default region when setting up a service but it has capabilities that the other regions do not. As a result, even if one’s AWS deployment is distributed over several regions, depending on the complexity and needs of the app, it may require EC1 for a tiny bit of functionality which, if absent, breaks the app.

@Mer__edith (For me this unexpected outage was quite dramatic, as it happened at exactly the time my son was going into surgery and we communicate a lot via Signal, so my "best wishes!" didn't go through and his "All went well" arrived with a significant delay. Murphy, I know. Things go wrong when you least expect it. He is fine and all is good :)

@toxy And that's what SRE (Site Reliability Engineering) should solve — to have a (temporary, even manual for these very rare cases) failover solution to, if needed, a completely different provider, in case that one provider goes down. That is my hope. That even if this rare case never happens again, there should be a plan available. @Mer__edith

@jwildeboer @Mer__edith I suspect this outage has been a bucket of cold water for many organisations. The other obvious factor is cold-hard cash. I’m no expert but I suspect an iron-clad cloud-failover redundancy service at scale would be pretty damn expensive to the point of economically unviable for many outfits.

@Mer__edith @systemadminihater

There are two issues with that disaster scenario.

First - AWS has numerous healthy competitors including google's GCP and Microsoft's Azure and Oracle's OCI, and they are gaining rather than losing market share. Their immanent demise is (sadly) unlikely. So robust competition seems pretty assured. And even if those three disappeared, there are other wildly strong competitors (Cloudflare!) waiting in the wings.

Second - self hosting never went away, and at a certain scale it's not just viable but still beats AWS in many ways. Point being - a 5x price increase would mean a sudden mass exodus from AWS. But that's unlikely as well - AWS pricing is both consistent and downward trending, with certain weird exceptions.

The reality is there is no magical superior never-failing alternative to AWS. Shit happens. You do not succeed by never failing - you succeed by reducing the impact of the fails. Hosting on high-resiliency proven platforms like AWS is part of that strategy.

If Signal usage is a single-point-of-failure issue for someone - that's their problem, not Signal's.

@Mer__edith What if, instead of running a global comms platform for millions of people that requires AWS level infrastructure, we run a bunch of small, local ones that all federate and interop with each other? 😍

@Mer__edith Maybe we should focus more on cloud interop, to reduce migration costs and enable better market dynamics? Too bad, EU is mostly just wasting money with their sovereignty initiatives…

@promovicz what would you suggest the EU should do differently? They cannot force US companies to do anything unless it affects EU citizens, and Amazon's services aren't targeted at consumers. On the other hand, the DMCA makes reverse engineering a felony

@punissuer

States have no obligation to be fair with monopolies.

* regulate their prices (consumer or otherwise)
* withdraw their advantages (tax, market, regulatory)
* use cartel law
* use state money against monopolies

maybe:

* reintroduce our DMCA waivers
(Germany had a law for it)

@yawnbox @Mer__edith Try running video calls over Tor.

@davep @yawnbox @Mer__edith Pretty sure that wouldn't even be as big of an issue as long as you don't try to exit the network.

You could even potentially improve the throughput ability by making every client that wants to use the network a node that relays traffic when it doesn't have active calls, however that's not suited to be automatically activated on mobile devices with limited power or even data caps. (But I would imagine people would be willingly donate resources to such a network if a simple separate application was offered the same way as it's done with TOR already)

@toxy My fear is that many organisations will shrug this off as "too rare to prepare for" event :( I hope Signal can help here by sharing a possible solution (should they find one, of course) @Mer__edith

@jwildeboer @Mer__edith I’m not sure. The big cloud providers have been laying off costly (experienced) staff left, right and centre so I suspect these outages will become more common in short order. It’s not that the current staff lack the knowledge per se but that lack of muscle memory and hands to the pump will greatly increase downtime in what are in many cases, legacy systems.

@jwildeboer @toxy @Mer__edith You need to run the failover solution all the time, because if it isn't life tested, it will very likely not work.

So you better design for at least a federated system. That's a much better design decision, anyways.

@yawnbox @Mer__edith Not once the number of active users approach even a fraction of number of active Signal users, without vastly expanding number of exit nodes etc. That could happen but it requires large scale benevolent effort and resource expenditure.

@troed @Mer__edith Than you, obviously, should have no problem setting up an alternative with the same features and sizing as signal 👍

@gilbus

It's called Matrix and yes I run a Matrix instance.

Did you have a point?

@Mer__edith

@Mer__edith @JadedBlueEyes @troed in reality the hyperscalers barely register. See https://fedidb.com/stats and https://blog.benjojo.co.uk/post/who-hosts-the-fediverse-instances for more details. Hosted from people's homes likely outnumbers all of the hyperscalers combined.

@jonathan @Mer__edith @JadedBlueEyes @troed that doesn’t change the fundamental point though; you can run a mastodon server on a raspberry pi but you cannot run something like signal without using at least 1 of about 3 or 4 american tech companies

@repeattofade

I don't know if we're having the same discussion. No, you cannot run a centralized service like Signal without doing so - but since no one is claiming that either I'm not sure what your point is.

You can however run something like Signal that's decentralized. We know this, since it exists. It's called Matrix, and many people in the Fediverse also run Matrix instances.

@jonathan @Mer__edith @JadedBlueEyes

@Mer__edith not so surprising, as it's very difficult to do anything at any scale online and avoid AWS entirely.

The surprise shouldn't be about Signal, it should be a rallying cry to build diverse infrastructure.

@Mer__edith

Mastodon? The comparison is flawed.
You can run Mastodon without AWS, etc. just fine.
You even can run an instance in your basement if you want.

@m_berberich
@Mer__edith
Could it be that by now other messaging services are now **more #privacy‑friendly than Signal**? That’s what’s being claimed here: https://privacyspreadsheet.com/messaging-apps

And, based on Meredith’s comment, would a platform like **SimpleX**—if it ever matched Signal’s user base— also have to rely on **AWS**? 🤨🤔

@troed @jonathan @Mer__edith @JadedBlueEyes I feel we all understand the landscape, what exists and why

this just feels like a really unproductive, unhelpful thread from foss decentralised self-hosting absolutionists, trying to say they understand how to run signal better than its president

use matrix, by all means; try getting your family and friends to use it too. good luck. “perfect” really can be the enemy of good (or in this case, privacy).

@repeattofade

So far it seems Meredith does not know how decentralized service like the Fediverse and Matrix work (the claims that most use hyperscalers). No one is however claiming that we know how to run _the centralized service Signal_ better. We're saying maybe don't be centralized.

"try getting your family and friends to use [Matrix] too. good luck."

Thanks, yeah, the whole family does - including my elderly parents. It seems you might not know the subject you're having opinions on?

@jonathan @Mer__edith @JadedBlueEyes

@daniel @Mer__edith Even _IF_ it were possible to create a black box version of "distributed Signal mesh node in a box" that you could run in your basement to help make Signal more tolerant - I mean with enough $ and willpower Im sure it could be done - there's still the question of: if you don't control physical access to the node, there's still potential for attack regardless of how much encryption and protection. Would you ever be able to trust it completely?

@tezoatlipoca @daniel @Mer__edith with global hyperscalers like AWS you don't exactly control who has physical access to your node either. You simply trust in Amazon holding up to their promises 🤷

@Mer__edith Does not answer the most basic issue.

Why the hell signal fail if us FAIL.

IT's simply you that messed up.
Also magnificent lie on how "only aws exist", like you are "forced" to use everything from aws and not use multiple one in multiple country.

Took nearly a week for this answer. god resign and let somone else take over.

@Mer__edith @buherator so much that 1 dns issue and the whole network is done, when you totally stable president will say to bezoz kill signal i'm sur you will survive ....

@Mer__edith At least for messaging I found that 30s latency are acceptable to most. People no longer expect their tools to be fast, and they type slowly.

That does not solve video and audio; WebRTC may help.

One step forward could be to change education so that more than a few handful people understand how peer to peer networks of the past managed to connect millions of people with a handful of servers and ISDN bandwidth.

My humble contribution:
https://www.draketo.de/software/p2p-talk.pdf
https://www.draketo.de/software/p2p-talk

@Mer__edith ⇒ push functionality out of the cloud bit by bit, as people re-learn what works and what fails.

Experience with that is scarce, though. Interest seems quite big, but polluted by crypto-coin agenda.

Anyway: thank you for your writeup!

@ArneBab @Mer__edith they are american, tech bro, they expect to have their cat picture in 0.000003s, it's so dumb, even call can have an 1 or 2 s lattency.

@lexinova do they actually expect that?

Don’t they have at least 200ms of transitions and animations before they even see the message?

I’ve been communicating with IRC over Hyphanet for a while now that has 30s round-trip-time, and it’s strangely barely noticeable.

The main issue may be the notification "server received your message" (✅ ✅) and the info "other side is typing".

@Mer__edith

@ArneBab @Mer__edith For me her answer can be 2 thing :

1. Corpo bullshit to calm userbase,
2. US fried brain that lost contact with reality on how half of the world don't have a low lattency network and everything work anyway.

@lexinova Hosting infrastructure for multimedia communication of 100 million users¹ with just 50 employees¹ actually is hard.

¹ https://backlinko.com/signal-stats

So I do not think your points apply.

@Mer__edith

@ArneBab @Mer__edith Did you remember that actually, only people who enable "relaying for call" would need this low latency ?

because by default, signal try to connect directly to the other user and relay if it fail.

As such, in most normal case, this low lattency won't apply at all.

and for picture / video even if it take 20s instead of 10 to download on your device what will it change ?

@promovicz @Mer__edith You mean something like https://sovereigncloudstack.org/ ?
The intial phase was also financed by the EU and German government ;)

@davep You can run video calls over jitsi.

IPv6 was supposed to solve NAT, so fewer servers would be needed. Supposed to.

@yawnbox

@ArneBab @davep @yawnbox Note that in the specific use case of Signal: given their threat model, "direct peer-to-peer connections by default" are not desirable. You'll need to bounce the audio&video traffic by default to make it more costly to infere who is talking with whom.

So the fact that working NAT and IPv6 help rely less on TURN servers won't help decentralize that much.

@dryak Maybe a start could be to switch to direct peer-to-peer connection if Signal sees that both sides are in the same subnet (i.e. on the same wifi).

In that case they connect to the signal server for connection with a voice-data profile *at the same time* which already gives away that they are talking, so staying in the subnet with a direct peer-to-peer connection would reduce the total privacy loss.

@davep @yawnbox

@dryak but firstoff, to stop discussing with too little information: the reason they use a forwarding server is that a single device can’t send video to 40 people via direct connections.

Here’s their description: https://signal.org/blog/how-to-build-encrypted-group-calls/

That also shows the level of complexity involved already.

@davep @yawnbox

@Mer__edith does that mean that it could be made possible to fallback to a system where messages still get through, just not in real time?

@Mer__edith

i'm more concerned that signal do best practices in terms of redundancy/failover and data encryption for cloud. it's quite possible to do cloud (or multi-cloud) robustly but it takes extra effort/engineering/expense over more minimal uses of a cloud service.

can you give any details of your multi regional, multi-cloud use, active/fast failover, etc.?

@Mer__edith

i love this goal but you've done a great job of laying out why it's really hard, really expensive, and daunting enough that even the EU is struggling with it.

protocol changes can help but at a certain point, you hit speed of light and i don't think we're quite at the stage of engineering around that yet. :D

thanks for this thread. good to discuss these things, not just lay blame on a provider failure.

@fiery @troed #AWS *is* necessary if you're trying to make a large-scale centralized system, no? There aren't a lot of alternatives to that when we're talking about the level of scale that #Signal operates at.

@realcaseyrollins@noauthority.social @fiery Not only aws is not required to build a large scale centralized system but Signal is also is not necessarily required to be centralized.

@fiery @Mer__edith

Interesting. I'm not aware of how many #AWS competitors there are so maybe I'm wrong.

That said, I don't disagree with you that #Signal shouldn't be centralized, it's one of the reasons I don't think I've ever used it.

@realcaseyrollins@noauthority.social @fiery Aws is not doing magic. Your compute still have to run on physical machines and the ones they have are not special. They also do not have to sit on Bezo's datacenter for large scale systems to work. Now if you are talking about their distributed systems' architecture, now it is not about centralized systems anymore, is it? Cloud is just a magic word that means other people's computers.

this post in particular is utter horse shit

@Mer__edith So you know don't forget how mozzila died. like you know they took decision uppon decision against their community and died.

The community is angry that an US DNS can down nearly all the network, and you simply tell us you will do nothing.

If signal continue on this path ... you could go with mozilla in the same building i guess.

There certainly are cloud competitors to AWS. How easy it would be to use them would depend on what services Signal uses in AWS. Some will have equivalents, some may not. AWS, being around for so long has a boatload of services and it’s not in their best interest to make them easily movable.

But I absolutely get why something like Signal would use a cloud provider. Could it be done entirely on-prem? Quite probably. However could they do it within a business model that would allow the scale of users to use it as they have today without charging significant fees to use it? I highly doubt it. This would hold true for anyone wanting to build a service like theirs that would operate on the their scale. The bandwidth and other infrastructure would be immense and super expensive to buy and maintain. The only folks able to provide that would be big telco, tech companies.

Could it be all decentralized ala the Fediverse? Sure and such services exist. But, much like the Fediverse, getting user adoption would be much more difficult and tour audience would be those tech savvy enough to use what’s already out there. I mean, for example, Matrix/Element exists. Quite secure, very decentralized. But it’s not for the general public.

@midway@soapbox.midwaytrades.com You are comparing "Cloud" (in this context actually meaning PaaS) to owning your own datacenter as if they were the only options possible, but there is a whole world of options and combinations of options that are not either of these two extremes. I keep getting surprised that so many people actually believe that owning your own datacenter and equipments is the only alternative to so-called "cloud" providers.

@ArneBab @davep @yawnbox there's lots of alternatives that don't need the big players - but not for organisations that need to scale like Signal *and* are pretty small themselves.

Signal is and has always been a tradeoff between ease of use, onboarding and security/reliability.

And especially in an activist context both are important. Both are also important in order to provide secure communication to the masses.

But yes, it's also advisable for communities with any tech and org skills to create backup communication because centralisation always also creates power imbalances and threats.

But currently I'm happy if I can activists off Telegram and Discord 🙄

@Mer__edith To clarify, are you multi-region within AWS? A bit hard to find info on this.

(A little concerning that so many people are worried about the privacy implications of Signal running on AWS... maybe some missing education on E2EE.)

@varx @Mer__edith clearly not, they messed up their infra and say it's AWS.

The only fact that US downtime lead to a worldwide downtime of signal show their system is not well done at all.

@Mer__edith but its also concerning that a failure somewhere in the US brings down Signal users in Switzerland, Turkey , Iceland, Sierra Leone and Nigeria. Yes hyperscalers are nice as they are everywhere but the dependency becomes brutal and expensive. Im running myself a backbone across multiple continents and redundancy is an absolute must. Signal has failed here. Blaming only AWS is the wrong path. Signal is big enough to do it better. Thats why I am a long term subscriber of it.

@afink @Mer__edith i definitly lost trust in signal.

because in my bood the fact they had a downtime is not the issue.

The fact they blame AWS, are condescending over the one that ask why it happen, and do absolutly noting to fix the issue on the other hand, tell more about signal ... than any bug or PR com.

@forthy42 @jwildeboer @toxy @Mer__edith and nothing prevent them to use the federated one as failover, and allow the user to force the federation if they want.

With federation they would reduce their need, reduce the price of running signal, and having federation as faillback when AWS crash again.

@Mer__edith @danstowell funny how you lie so much by omiting many possibility that would allow both AWS and federation (running at the same time), Seem you are just a fake that lick the ass of big tech while "fighting" against them in public, resign and allow true people to find a solution that work.

@Mer__edith I'm sure millions of folks appreciate your service and the work you all do. but I think that 2025 Oct 20 outage revealed Signal has a de facto SPOF on AWS us-east-1. its an architectural flaw not unique to Signal, obvs, since it bit lots of other systems around the world. I do hope your team can devise a way to change your architecture to remove that SPOF

@synlogic4242 @Mer__edith it also show the hypocrit nature of signal, saying they fight big tech, while licking their ass to maintain their server.

@Mer__edith @buherator wha you use multiple, so much to say google and amazon.

Google was useless to mitigate, and the world failed due to an US server failure, that show your design is failed.

I would fire the guy that allowed that if it happened in my company.

You, you do nothing.

@Yuvalne @buherator @Mer__edith it's an appologic message.

GCP was used, maybe, Signal failed ? definitively.

Result, it was useless the DRP failed fire the guy you made it.

Sorry but a it helped is not good enough

@geofurb @Mer__edith

Use a privacy respecting keyboard then. But fair point for sure, especially for newbies

@Rhababerbarbar @geofurb @Mer__edith on many device it do not exist (aka iphone) where apple one is the most private.

@Mer__edith The even worst part imo is that usually when a smaller player comes into play and they have LOTS of brains and rare talent and passion and love for what their doing, will either get lost because people's everyday life is soooo dependent on these evil corps. OR these corps. will buy them out... BUT that means that more effort needs to be put in decentralization, as way of life of sorts. As a stand. Of stop using one thing for everything and instead try to have more things that do ONE thing and do it well. Decentralization = more control from the person and less control from the big players. Centralization is the exact opposite... But it's easy... :/

@nakdim @Mer__edith and you have tech bro like signal, where they know better and ignore the feedback when something like this happen and user are angry they whinne it's not their fault and do nothing to prevent it for the future and will repeat when the next issue will arrive.

@Mer__edith I understand what you're saying but I also wonder if maybe you've been sold a bill of goods. I run a real time communications company (admittedly very much smaller) and one of our core philosophies has always been data sovereignty. Of course as you rightly noted we can't build data centers but there's plenty of space for lease globally where you can scale you're own gear. We've made access to the physical layer a priority, maybe Signal can as well?

@mike Does it run on donations at your scale?

@lexinova Not necessarily. I recall something about AWS *itself* having some things still centralized in us-east-1. And Signal might rely on other vendors that aren't multi-region. You get the idea.

I'm honestly fine with some downtime here and there. I think people should be more chill about it in general.

@varx i'm not chill about it because they serve us fallacy and lie, and do nothing to prevent it to happen again, if they had say "we screwed up make so it don't happen again" i would have said nothing, but no here we had the corpo bullshit and a condescending answer.

@lexinova Realistically, they have to make tradeoffs about how they spend their time and energy.

They're not saying it here (and it would be nice if they did!) but they could choose between multi-cloud resilience or doing product development. I think most people would prefer the latter.

@varx here for me, it's not a tradeoff, they just try to ignore it and do nothing about it, saying hyperscaller is a lie, my company use hyperscaller, but unlike the liar of @Mer__edith you can combine them, by having US one in USA, EU one in EU, CN one in china etc.

they put all their eggs in the same basket and now they lie their teeth out to make us thing it's not their fault.

@varx Also @Mer__edith try of deflecting the responsability show she's unfit to be the ceo of @signalapp as #CISO of my company i had to appologize and take the blame for multiple time where partner screwed up.

because when you are the chief you are responsible

AWS failed, she's the signal chief it's HER responsability, so her fault.

if she cannot accept this, she's not a good CEO and must resign.

@varx @lexinova

This is correct. Besides, taking down a big (the biggest) part of a system necessarily compromises what remains.

@jdormansteele2 @varx but for many of us again it's not the fact the issue happen that make us angry.

it's the fact signal do not take it's responsibility and blame their partner, as i said in another post the chief of a product is always responsible, and blaming other won't change this.

Their reaction to this failure disapoint me even more than the issue itself :

1 week to adresse de issue.
2 when adressing it blaming partner
3 be condescending on other and lie on hyperscaler necessity.

Theses 3 point are what piss me off ... not the fact Signal failed after AWS

@simonzerafa @ericfreyss @Mer__edith
i do want to mention something important, which is that Signal doesn't only use AWS, and actually the fact they don't was key to them recovering faster than AWS did by rerouting users. however when the issue on AWS is as fundamental as the DNS for DynamoDB being broken, there's gonna be downtime for some users, there's not really a way to go around that.

@Mer__edith

looking at your replies to replies here that seem to make sense to me (especially re decentralization), you're telling them they don't know what they are talking about. well I definitely don't.

like with debates re #ATproto and #ActivityPub, I have thoughts but I know that we really need to see the experts debate each other more somehow. I don't think it happens enough. so I'd say the same re #signal and #matrix etc.

@wjmaggos meredith gets a lot of replies that could be answered with a little bit of research or could be answered by anybody, which may explain the short replies; but i can try to answer you

@Mer__edith

@wjmaggos what meredith is explaining is the scope of the infrastructure that signal absolutely needs in order to provide the service it does at the quality it requires, and that there are extremely few options that satisfy those needs

to say building their own infra would be prohibitively expensive wouldn't begin to describe it

@Mer__edith

@wjmaggos signals threat model and security guarantees can't be met with decentralization like matrix, fedi, or atproto, either. when disparate servers communicate, they have to know how to relay messages between each other, which leads to a lot of metadata leakage (as is the case with matrix)

tor likewise has mitigations for time based correlation attacks, which are great for its use case; but would cripple signals quality

@Mer__edith

@wjmaggos i agree that it sucks to rely on amazon (or google or microsoft), but its the reality of internet infrastructure right now

it's relatively true that there are some attacks one could leverage in this scenario (particularly "harvest now decrypt later" attacks), but signal was very quick to adopt post-quantum cryptography which should provide a pretty good confidence that those attacks aren't feasible for the foreseeable future

@Mer__edith

@xyhhx @wjmaggos @Mer__edith ewwwww signal shill ewww hey fedi cancel this user

@soop @wjmaggos @Mer__edith @xyhhx just say you don't have friends and keep it pushing dog

@repeattofade @troed @Mer__edith @JadedBlueEyes those are great arguments, but the one Meredith is taking on this thread is "look, Mastodon can't/doesn't do distributed infra" when the data says that actually yes it does.

I actually don't have a strong view on the centralisation of Signal, but seeing Meredith talk down to people (see replies on this and other threads) and then be wrong on the facts is pretty galling.

@mrRobot
@Mer__edith
Not every service has to implement every best‑practice—users have different threat models. For me, **Perfect Forward Secrecy (PFS)** is essential, and it’s missing from both Threema and iMessage. Likewise, **client‑side scanning** in iMessage is a hard‑no. Without PFS and with on‑device content analysis, the communication isn’t private for my needs.
In the past, Signal was the messenger with best‑practice safeguards

@jack @Mer__edith I’ve actually been using #SimpleX recently myself in addition to #signal for conversations where I want the strongest privacy posture. It’s doing a lot right: strong FS properties, reduced metadata exposure, and no central identity system. So I’m not arguing from inside Apple’s walled garden here.
Though I still get uneasy when #privacy gets reduced to a binary grid of checkboxes. Private enough always depends on who the adversary is, what they want, and what the people you’re talking to will actually use. Cryptography isn’t the only variable in the real world.
On #Apple/iMessage: the original client-side photo-hash plan was a real problem, though that rollout was scrapped. The current “communication safety” feature for minors doesn’t break adult E2EE. It might evolve into something dangerous later, but at the moment calling it disqualifying feels more like a principled fear of direction than a present-day compromise.
#Threema’s history with limited PFS and audit transparency is worth criticizing. Though if they’ve now implemented PFS across the board, I’d prefer to see more independent scrutiny before writing them off as perpetually “not private enough.” Once again, context matters.
SimpleX gives stronger guarantees for high-value targets, sure. Yet adoption friction introduces its own kind of risk. If I push everyone I know into tools they won’t stick with, they’ll drift right back to far worse defaults. Usability failures can erase all the theoretical privacy wins.
So my stance looks something like this:
• Use the strongest tool that your social graph will realistically sustain
• Treat all vendors as temporary allies, not saviors
• Demand transparency and scrutiny from whoever holds the keys
• Always ask “private enough for whom?”
Security is a negotiation with reality, not a purity contest. You and I may have similar goals and even use the same tool for sensitive chats. We just assign slightly different weight to the tradeoffs.
Happy to keep comparing notes as these apps evolve. Yesterday’s “maximum” becomes tomorrow’s minimum pretty quickly.

@mrRobot
@Mer__edith
> Demand transparency and scrutiny from whoever holds the keys

Is it really possible to achieve full infrastructure transparency for a Contact Discovery Service? Can we actually verify that it runs inside a protected AWS VM (with eg ORAM) and not just an untrusted execution environment?
Considering that social graphs are a **high‑value target for intelligence agencies**, such verification would be crucial. Or am I wrong?

#UEE #AWS #NitroEnclave #Trust

@troed @Mer__edith hopefully the pros and cons in this comparison could provide useful context:
https://hapyyr.com/@bogo/115401249466782443
But ultimately this is not a conversation about features. It is about priorities. And as already said on this thread, decentralization is not compatible with low latency.

@mapto

I've indeed seen several people claim that low latency requires centralization, but I don't think I've seen anyone show why that would be true.

My (way back when) background is in telecom and that has always been low latency and decentralized.

@Mer__edith

I've been using #Signal from the very beginning (TextSecure times), and I've been advocating Signal a lot.

But the centralized architecture, instead of a federated decentralized approach is something I never liked. Also the focus on BigTech platforms (IOS, Android) is something I do not like. I'm using a #Librem5 #Linux phone, but there is no official primary client for Linux.

Still, I'm donating, but I would appreciate addressing centralism and #BigTech dependency.

#MobileLinux

@janvlug @Mer__edith

I don’t believe federation is possible without leaking a huge amount of metadata, but I believe decentralisation is. It would be great to see Signal actively investigating the options in the design space that enable it.

The client side is a bigger problem. The choice of license has two major impacts:

• Only Signal (who have a CLA and so are not bound by the license) can publish anything on the Apple App stores. This means that a legal injunction against Signal shipping the app in a country will cut off around half of the users in that country.
• No one can integrate Signal with their back end systems. A load of people install WhatsApp or WeChat because some institution uses them to communicate with customers and then uses them to talk to friends because they’re already installed. I’d love for my bank to be able to communicate with me via Signal instead of POTS, for example, and then for Signal to benefit from the network effects: if the bank says ‘install Signal to communicate with us securely’ then that’s a few million folks who will have Signal installed already.

It is very rare for a protocol to become ubiquitous without a permissively licensed reference implementation.

I didn't mean to put it that way. I mean peer-to-peer is certainly a thing. And we have systems that do that....and they are WAY too complex and cumbersome for the average user to use...see Matrix as a classic example. Quite secure, very decentralized, but not simple enough for most people to use. Heck, even here on the Fediverse, the user base is quite limited because of the decentralized nature is just too much for most folks to grasp..throw real privacy and zero trust encryption on top of it and your app will never take off.

Therefore, if you actually want users, you're going to have some amount of centralization. That means you need to run on something, either your own gear or someone else's. And at the scale that Signal wants to run, cloud makes sense not just for compute and services, but also the sheer amount of bandwidth needed to process the amount of data they want to send.

Can it be done a different way? Sure. Will those methods scale to the reach the average user? I seriously doubt it.

@midway@soapbox.midwaytrades.com Now you are talking about something else completely. You are making the point that centralization somehow improves UX. You'd have to substantiate that better.

@troed would it sound better to you if we substitute "requires" with "is easier with"? At least from my point of view this shifts the burden of proof to you.

@mapto It's not like I'm talking about some unknown magic here.

https://social.woefdram.nl/item/a6a9ef48-97a5-40d0-a953-4dafcd367253

Yeah, well the conversation has several branches.

Centralization simplifies how thing work in general, especially for end users. You have one place to go where you set up your account and work from single experience. There's a reason why every successful service our there has some level of centralization. It's just easy to use. Ease of use beings in more users which helps the service survive.

Decentralization has some great advantages. But with that comes complexity and with complexity comes a lack of adoption. The lack of adoption means that there's no money in it. And that's great if you're a hobbyist, but not if you're a company.

An easy example is social media. Look at all of the massive services. They are all centralized. Look at a decentralized system like the Fediverse. Yes, it's very decentralized, but the audience is very limited.

Now let's take this back to Signal which was the whole point of the thread. Yes, it has some centralized services. Those centralized services make the system work well enough that average internet users would actually use it. There are decentralized options out there. They work peer to peer so there's no need for things like cloud infrastructure or a big data center to run them. Matrix/Element comes to mind. Super secure, decentralized messaging. Very few people use it because it's just too complicated for the average or even above average user.

So if I'm Signal, a company that wants to build a more secure messaging app, I'm going to make some compromises in order to make it acceptable and palatable to a wide audience so I have a chance to make some money and keep my companhy afloat. Thus, something like AWS makes sense. I can get access to huge resources to handle any user load, but my costs scale in real time with my usage. This is sensible. But there are trade-offs. But i think for what Signal is trying to do, those trade-offs make sense.

@midway@soapbox.midwaytrades.com You are mixing many different concerns here. First, we got define better what we are talking about when you say "centralized". That can be many things. The user does not have to even see this to use a service. Case in point: facebook, twitter/x, tiktok and other are ALL decentralized services. In the sense that they are distributed systems split in many parts, running across many machines. How do I know that? Because no computer exists nowadays that could run them on a single machine. Yet users are kept blissfully unaware of that as they should. Signal is ridiculously small compared to those social media. There is no reason a messaging service like signal could not be the same and yet be decentralized internally. Even amazon itself is not internally a centralized system in any sense of the word. They are highly distributed internally and offer plenty of options for redundancy. And yet Signal was down when ONE REGION of aws was down. That kind of centralization serves no purpose and is just bad engineering. No one would be harsh at signal had they owned up to it and said, "yeah, that was bad, we need to do it better". But no, their leadership went on condescending everyone, telling that they do not understand the problem space. That was just bad. Mind you, signal is still my primary communicator and I still donate to them monthly, while I am still using it. But when a CEO earning upwards of 700k usd a year gives that kind of response to the public, that is making me reconsider. Trust has been lost, something is off.
Now another point is that non-centralized does not necessarily means peer-to-peer. One such highly successful example is email, which is federated. Yes, most users will just gravitate to some centralized offering like gmail or hotmail, but the system is still interoperable for folks or companies who want more control or even self host. We have options, based on public standards. In that sense even instagram is being more open than signal, in the sense that they now have threads which talk to the fediverse. Signal is openly against any such federation arrangement, thus reducing the power that users have over their own data. They do not even have good export options, arguing that would reduce security. Yet they require a mobile number to sign-up which in most places already doxx the user.

@yawnbox @Mer__edith How about a North American Signal, and a European Signal?

Those two could federate and if a government on either continent goes bonkers, it can at least not take down all of Signal.

If you are defining a centralized service as one that runs in a single system, then this has ceased to be an adult conversation, especially here on the Fediverse.

I get only running in one region is a vulnerability. It could be bad engineering…it could also be because of cost. Resiliency isn’t free or necessarily cheap, especially for a company that relies on donations. It’s great that you donate to Signal but I assure you the vast majority of their traffic is sent and received by people who don’t.

I made the point about running in the cloud or on prem because that was part of the pro original post (at least as I remember it…it’s been a while). The email model is essentially peer to peer. It relies on lots of places agreeing on a standard to send messages. The issue with this is that to make that work requires dumbing down the standard and would likely break the goal of an all like signal. Email is not in any way secure. Quite the opposite in fact. Are there ways to make it more secure? Yes. But there is no agreed to standard to do so and thus this feature has not been widely adopted. The way email has gone is to become more and more centralized every day with a handful of companies providing email whose business models do not want secure email. The email market has decided that free is better than secure. The price of free is the provider reads your email to sell your information. I only went down this rabbit hole because Signal won’t want to adopt this model because doing so kills their entire reason to exist. Their compromise is that they handle and procrss the

@midway@soapbox.midwaytrades.com Here are the definitions I am using for decentralized, distributed and federated: https://networkcultures.org/unlikeus/resources/articles/what-is-a-federated-network/

@lexinova

Fair point.

I would say creating a privacy respecting keyboard app would be better then?

Thats like using a #VPN only in the browser, or adding #FreeSoftware apps on a system full of #spyware... doesnt make much sense and you leak more data than you think.

@Rhababerbarbar i'm using graphene so my keyboard is safe, but i agree moving to another os is the priority in those case.