Enhance KU Publication: Topic Handshake & Outbox

by Sebastian Müller 49 views

Hey guys! Today, we're diving into how we can enhance the publication of Knowledge Units (KUs) by implementing a topic handshake and leveraging an outbox discussion category. This is a super cool way to ensure our messages get delivered, even when things aren't perfectly synced up.

Why This Matters

Let's talk about the why behind this enhancement. Currently, we face an issue where PublishError.NoPeersSubscribedToTopic can block the publication of KUs. This happens when there are no active subscribers at the moment a message is sent. Imagine sending a message into the void—not fun, right? Our goal here is to avoid this scenario and make sure every message finds its way to the intended audience, even if they're a bit late to the party. This is crucial for maintaining a robust and reliable communication system. We want to make sure that no message gets lost simply because there wasn't an active subscriber at the exact moment of publication. By addressing this, we're ensuring that our system is more resilient and user-friendly. Think of it as setting up a reliable postal service for your digital messages; you want to be sure that your letters are delivered, even if the recipient isn't home when the mail carrier first arrives. This enhancement is all about creating that reliability and ensuring a smoother, more seamless experience for everyone involved.

What We're Doing

So, what’s the plan? We’re implementing a system where messages published without subscribers go into an outbox. Think of it as a temporary holding area for your messages. As soon as a subscriber pops up, these messages are automatically flushed out and delivered. This ensures that no message is left behind, even if the timing isn't perfect. We'll also be adding clear logs for debugging the handshake flow, so we can easily track and resolve any issues. Plus, we’re setting up end-to-end tests with two nodes to simulate a late subscriber scenario, ensuring our solution works flawlessly in real-world conditions. To prevent any spamming, we’re implementing an exponential backoff on retries and rate-limiting. Finally, we're introducing a handshake protocol on sgn.ctrl.hello.v0. This is like a secret knock that ensures everyone is who they say they are before any messages are exchanged. By implementing these measures, we're building a more robust, reliable, and user-friendly system for KU publication. It’s all about making sure our messages get where they need to go, when they need to go, without any hiccups along the way. This holistic approach ensures that we're not just fixing a single problem, but creating a more resilient and efficient communication ecosystem.

Success Criteria

How will we know if we’ve nailed it? Great question! We have some specific acceptance criteria to ensure we’re on the right track. Imagine a scenario where Node A publishes a message, but Node B connects later. Our goal is that sgn publish --file ku.json doesn’t fail. Instead, the message should go into the outbox. Then, when Node B connects, it should automatically receive the message from the outbox. This simulates a real-world scenario where subscribers might not always be online at the exact moment a message is sent. By ensuring this process works smoothly, we're guaranteeing that our system is robust and can handle various timing scenarios. Think of it as testing a delivery service: you want to make sure that packages arrive safely, even if the recipient isn't home when the delivery person first arrives. This acceptance criterion ensures that our system functions reliably and efficiently, providing a seamless experience for all users. It’s about building confidence in our system and knowing that it can handle the complexities of real-world usage.

# Scenario: nodo A publica, nodo B si connette dopo
sgn publish --file ku.json  # non fallisce, va in outbox
# nodo B si connette
# nodo B riceve automaticamente il messaggio dall'outbox

Technical Deep Dive

Alright, let’s get a bit more technical. Under the hood, we’re employing a subscribe-before-publish pattern. This means subscribers need to be ready and listening before messages are sent, which helps prevent those frustrating “message in a bottle” situations. We’re also using an outbox with a Time-To-Live (TTL) to prevent memory leaks. This is like setting an expiration date on messages in the outbox, so we don’t end up hoarding messages indefinitely. Think of it as cleaning out your digital mailbox regularly to keep things tidy. To further optimize the system, we’re implementing rate limiting on outbox flushes. This prevents any one subscriber from overwhelming the system with too many requests at once. It’s like controlling the flow of traffic to avoid gridlock. These technical measures are all about ensuring the system is not only reliable but also efficient and scalable. By addressing potential issues like memory leaks and traffic overload, we're building a system that can handle a growing number of users and messages without breaking a sweat. It’s a comprehensive approach to system design that focuses on both functionality and performance.

Key Technical Notes:

  1. Subscribe-before-publish pattern: This ensures that subscribers are ready to receive messages before they are sent, reducing the chances of messages being lost.
  2. Outbox with TTL: The outbox stores messages temporarily until a subscriber is available, with a TTL to prevent memory leaks by discarding old messages.
  3. Rate limiting on flush outbox: This prevents the system from being overwhelmed by too many requests to deliver messages from the outbox, ensuring smooth operation.

In summary, this enhancement is all about making our KU publication process more robust, reliable, and user-friendly. By implementing a topic handshake and outbox discussion category, we’re ensuring that no message gets left behind, even if the timing isn’t perfect. We're not just fixing a bug; we're building a better system for everyone!