Queues - Patterns of communication

• Two patterns of communication:
  • Synchronous (application to application)
    • Can be problematic if there’s sudden spikes of traffic
      • E.g. if you need to suddenly encode 1000 videos but it’s 10?
    • Better to decouple your application with queues for scalability and resiliency.
  • Asynchronous / Event based (application to queue to application)
    • Decouples applications
    • Services can scale independently from applications
    • 💡 Exactly-once delivery
      • One of the hardest things to achieve in distributed systems.
      • Better to design your applications to be idempotent: not to be affected adversely when processing the same message more than once.

Comparison

• SQS vs SNS
  • Choose SNS for if e.g. someone uploads picture and listeners need to different stuff (send thank you email, crop thumbnail & save in S3).
  • Choose SQS for application to application communication.
  • Fan out: SNS to SQS to listeners = good of both worlds.

• SQS vs SNS vs Kinesis Streams vs Amazon MQ

  Attribute	SQS Standard	SQS FIFO	SNS	Kinesis Streams	Amazon MQ
  Type	Direct queue	Direct queue	Pub & Sub (with topics)	Data ingestion	Direct queue + topic
  Consumption	Pull	Pull	Push	Pull	Pull + Push
  Latency	⭐⭐⭐	⭐⭐☆	⭐☆☆	⭐⭐⭐	⭐⭐⭐
  FIFO	👎 Best effort	👍 FIFO	👎 Best effort	👍 Per partition key	👍 Supports FIFO
  Delivery	👎 At-least-once	👍 Exactly-Once Processing	👎 At-least-once	👍 Per partition key	👍 Supports FIFO
  Retention	14 days, default: 4	14 days, default: 4	None	1 day (default), up to 7 days	Persisted in file until configured max file size reached
  Throughput / limits	None	300 transactions per second per API	Max topics / subscribers	Write/read per shard, can scale out shards.	Requires provisioning eg mq.t2.micro. Users, brokers, configurations and data storage is limited. API is throttled.
  Can delay	✓	✓	✕	✕	✕ Only on redelivery
  Out of box dead letter queue	✓	✓	✕	✕	✓
  AWS integrations	👍 Many	👎 Missing many e.g. S3/CloudWatch/SNS/Lambda DLQ…	👍 Many	👌 Some: Data Firehose, S3, Kinesis Data Analytics, Lambda, SDK uses DynamoDB for checkpoints	👎 Almost none, requires code.
  Consumer type	Same type	Same type	Different consumers can be different stuff with the event from the topic	Can be different per partition, as same consumer can listen to same partition	Can be same (direct queue) or different (topics)
  Use cases	Fast, decouple/integrate apps, visibility time out, message level ack/fail, message delay.	SQS + strict ordering, deduplication, decouple/integrate apps	Pub & sub (1 event to different apps), emails/SMS/.., push notifications, fan out (SNS to SQS -> good of both)	Analytics, logs, IoT.. Related records to same processor (partitioning), ordering, multiple apps to consume same stream concurrently, consume records a few hours later.	Lift & shift

Other services

• Amazon MSK: Amazon Managed Streaming for Apache Kafka (Amazon MSK)
• AWS IoT: Bi-directional communication between Internet-connected devices such as sensors, actuators, embedded micro-controllers, or smart appliances and the AWS Cloud.

---

---