irdest-corrupted issueshttps://git.irde.st/we/irdest-corrupted/-/issues2021-08-07T06:44:05Zhttps://git.irde.st/we/irdest-corrupted/-/issues/12Alexandria internals refactor2021-08-07T06:44:05ZKatharina FeyAlexandria internals refactorAs part of the NLnet grant to work on Irdest I allocated a milestone to get Alexandria to the `1.0` finish line. It's an ambitious target, seeing as the database is not yet [5 years old](https://twitter.com/sadisticsystems/status/942334...As part of the NLnet grant to work on Irdest I allocated a milestone to get Alexandria to the `1.0` finish line. It's an ambitious target, seeing as the database is not yet [5 years old](https://twitter.com/sadisticsystems/status/942334089578680320) (:wink:) but it will be important to be able to rely on storage in the future, and hopefully this will also make the `irdest-core/store` module obsolete!
Following is an outline of work that needs to happen. This issue will track development of these features.
## Alexandria: re-scope
As part of qaul.net Alexandria needed to handle many types of payloads. This is no longer the case! As such `blob` support needs to be removed. Furthermore, we will need to support large records, and possibly records where a key-value store approach is not ideal. Following is an outline of the **new alexandria scope**
* Encrypted data & metadata
* Identity concealing. Hide user IDs. Is this zero knowledge?
* Records based. Path based. Two types of records
* Key-value stores
* Table records with shared schemas
* Search tags. Encrypted tag cache. Per-user.
* Encrypted schemas.
* Insert data easily without manual diff creation
* Use diffs for transactions internally.
* Generational (epoch based) garbage collection
* Sync data dynamically.
* Break large records up into chunks that can be streamed from disk.
### Things that are stored
* Records! Chunk based.
* User sessions, with hidden user IDs. How?
* Per-user tag-cache
* Per-user private key store
* Per-user public key store (not encrypted - only user ID is unknown)
### The threat model
Hiding information from an adversarial user is impossible. Alexandria can not hide its own memory from the root user. Data MUST exist in un-encrypted form at some point or another.
Watching alexandria in memory WILL reveal information about data present in the database.
Protect data synced to disk. Confiscated devices, broken disk encryption, stolen files. Don't reveal user identity or information payloads to on-disk attacks. If the cops steal your phone, can they indict you?
## Internal overview
Following is a graph of data available via APIs, internal components and their on-disk counterparts.
![alexandria-overview](/uploads/efd08ca912588d6fba0a0dde8142690a/alexandria-overview.png)
## Outline
Following is an implementation outline
- [x] Investigate whether the `Encrypted` abstraction introduced in alexandria `v0.1` is still relevant
- [ ] Take a closer look at every component that exists in the code currently and map its inputs and outputs
- Where are modules being used?
- Are there unused modules in the library?
- [ ] Build a solid user session management module
- [ ] Allow user registration, login, logout, destruction
- [ ] Sync user sessions & keys to disk
- [ ] Tag & schema cache
- [ ] Encrypt caches with root user key
- [ ] Provide an API for other modules to query the caches
- [ ] Load information from disk that is not present in the cache (how?)
- [ ] Records store
- [ ] Investigate whether to split the store implementation between KV and tables or keep them in the same stare
- [ ] Create an API for manipulating tables & diffing KV records
- [ ] Create transactional deltas for the store
- [ ] Apply deltas atomically
Add and remove items from the above outline as needed.https://git.irde.st/we/irdest-corrupted/-/issues/11Room management logic2021-05-18T14:18:03ZKatharina FeyRoom management logicThe chat service fundamentally sends and receives messages via `irdest-core`, processes them in some form, and generates new events available via its API.
The basic room logic needs to be able to handle the following events:
* Manage r...The chat service fundamentally sends and receives messages via `irdest-core`, processes them in some form, and generates new events available via its API.
The basic room logic needs to be able to handle the following events:
* Manage room state
* Create a room
* Invite a new user
* Kick a user
* Set a room name
* Set a room avatar
* Send a text message into a roomLuxLuxhttps://git.irde.st/we/irdest-corrupted/-/issues/6Example service: chat2021-05-17T21:58:50ZKatharina FeyExample service: chatAs part of testing different routing approaches in Irdest, and to showcase the capabilities of the service API/ model we want to create a simple chat service.
The idea behind this service isn't to create a comprehensive chat application...As part of testing different routing approaches in Irdest, and to showcase the capabilities of the service API/ model we want to create a simple chat service.
The idea behind this service isn't to create a comprehensive chat application, but to create a simple example as to how people can communicate via Irdest, _and_ to create a simple entry point for newcomers to talk to people already using Irdest.
The requirements and development state will be tracked in this issue. Please refer to it in any change set that addresses the chat service.
## Design outline
`irdest-core` provides a service some encrypted storage, namespaced by user. This means that the service itself doesn't have to manage storage, and can do so via the irdest API.
Apart from that there are two main areas that this service needs to focus on: accepting user text messages and mapping them to irdest-core messages, and managing the state of chat rooms.
## Roadmap
- [x] Improve irdest-core storage API (#7)
- [x] Map service API to irdest-sdk (-)
- [ ] Add persistent storage backend to alexandria (#10)
- [ ] Create chat service API & SDK library (-)
- [ ] Room creation/ update/ deletion logic (#11)Katharina FeyKatharina Feyhttps://git.irde.st/we/irdest-corrupted/-/issues/5Binary distributions2021-04-04T15:16:06ZKatharina FeyBinary distributionsThis is a tracking issue for binary client distribution. As of 2021-03-02 there are three clients we may want to publish:
- [irdest-hub](clients/irdest-hub)
- [irpc-client](clients/irpc-client)
- [irdest-gtk](clients/irdest-gtk)
A ful...This is a tracking issue for binary client distribution. As of 2021-03-02 there are three clients we may want to publish:
- [irdest-hub](clients/irdest-hub)
- [irpc-client](clients/irpc-client)
- [irdest-gtk](clients/irdest-gtk)
A full irdest installation always involves `irdest-hub`, with some mechanism to make the daemon run in the background (systemd units, openrc, ...?), and one of the two clients. For most users the GTK client will be preferred.
## Tasks
- [ ] Build fully static Rust binaries (output an `.appimage` for each binary?)
- [ ] Provide release tarballs, including vendored dependencies for distribution packagers
- [ ] Create a small installer for full userspace installations on legacy Linux distributions
- [ ] What's the upgrade story? How does irdest learn of new versions?https://git.irde.st/we/irdest-corrupted/-/issues/4Alexandria API improvement2021-08-07T06:44:05ZKatharina FeyAlexandria API improvementThe `alexandria` API is causing some boilerplate in the `irdest-core` storage code. Some things are missing to improve the situation.
- [ ] Rename `.batch(...)` to `.batch_insert(...)`
- [ ] Add `.batch_update(...)` next to `.update(.....The `alexandria` API is causing some boilerplate in the `irdest-core` storage code. Some things are missing to improve the situation.
- [ ] Rename `.batch(...)` to `.batch_insert(...)`
- [ ] Add `.batch_update(...)` next to `.update(...)`
- [ ] Add `.upsert(...)` and `.batch_upsert(...)` as combinations of the above functions
- [ ] Provide a derive macro for `ToDiff`
- [ ] Create `alexandria-derive` (or `alexandria-macros`?) crate to let `ircore-types` depend on it, without having to depend on the whole database.Katharina FeyKatharina Fey