Skip to content

Draft: Ratman async overhaul

Katharina Fey requested to merge kookie/async-overhaul into develop

This milestone consists of a few sub-tasks:

  • A new client API encoding, based on frames and discrete command chunks
  • A new command line interface for ratctl and map it onto the new client API
  • Processing one chunk at a time. Each chunk produces at least one ERIS block worth of data. The chunk size should be chosen based on target chunk size and total message size. For example, a message producing 3 1K blocks doesn't need to be transported in a 8M chunk. Streaming a 10GB movie might warrant the 8M chunks however.
    • Encrypt the chunk with ChaCha20, the shared key derived from a diffie-hellman between the sender and recipient (public key address from recipient, and private key from keystore for the sender).
    • Create a block from the chunk and stream it into the journal
    • Slice the message into a set of carrier frames
    • Dispatch the frames along the most effective endpoint driver/ route
  • Investigate spawn strategies and default configurations. Following are a few questions:
    • Can we combine async "bundles" to allow running on truly single threaded (i.e. embedded) platforms?
    • Can we "split" an async environment to run across two cores, while sharing data structures between them?
    • How do we spawn the worker threads for large message transfers? How large does a message have to be for the request to be handled by a dedicated thread? Do we have a thread pool regardless? How do we handle varying sized of available threads to make sure resources are not underserved?
    • Etc... :S
  • Port the existing http stack to a tokio-backed framework (warp?)
Edited by Katharina Fey

Merge request reports