irdest-corrupted issueshttps://git.irde.st/we/irdest-corrupted/-/issues2021-08-11T21:16:02Zhttps://git.irde.st/we/irdest-corrupted/-/issues/23Use Kotlin DSLs instead Groovy Scripts2021-08-11T21:16:02ZGhost UserUse Kotlin DSLs instead Groovy ScriptsComponent whose code/documentation can be Enhanced
---
<!-- Select one out of code/documentation and delete the another one -->
Groovy scripts in application codebase, `build.gradle` files to be precise.
Proposed Enhancements
---
Usage...Component whose code/documentation can be Enhanced
---
<!-- Select one out of code/documentation and delete the another one -->
Groovy scripts in application codebase, `build.gradle` files to be precise.
Proposed Enhancements
---
Usage of user-friendly [Kotlin DSLs](https://kotlinlang.org/docs/type-safe-builders.html#how-it-works), that make the code readable and more understandable to the user. Also this is a step towards modernizing the application codebase, as a part of summer of '21.
If you're interested in Kotlin Scripts which we are going to use in our gradle files, see the [ref here](https://docs.gradle.org/current/userguide/kotlin_dsl.html#sec:scripts).GSoC 20212021-08-04https://git.irde.st/we/irdest-corrupted/-/issues/16Link Rust Library to Android Application Codebase2021-08-03T09:03:41ZGhost UserLink Rust Library to Android Application Codebase## Is your feature request related to a problem? Please describe
Currently, our rust library isn't linked to the application codebase, and application bears references of functions from the library and the library itself, as a result of ...## Is your feature request related to a problem? Please describe
Currently, our rust library isn't linked to the application codebase, and application bears references of functions from the library and the library itself, as a result of which it crashes each time it is launched. By far we've been using some _hacks_ to not initialize the library and just set the application build green and fix UI components. But now we wish to implement a robust and permanent fix that'll remove all the crashes that may occur due to incorrect lib referencing or linking.
## Describe the solution you'd like
Currently the problem which is encountered in library linking is that it fails to properly access/create a directory in android device to save state, and fails due to presence of lesser privileges, it fails to do so. Also, it turns out that the crate([xdg-rs/dirs](https://github.com/xdg-rs/dirs/tree/master/directories)) we're using for saving state across different platforms doesn't provide explicit support for Android(see [platforms](https://github.com/xdg-rs/dirs/tree/master/directories#platforms)). I also filed an issue there(see [xdg-rs/dirs/issues/54](https://github.com/xdg-rs/dirs/issues/54)) regarding the same but it went un-responded. Anyways, we fixed the problem internally at my local fork and hopefully will push the fix after cleaning up and optimizations.
The solution is to write a custom implementation for finding the private storage of the application in the device(which exists by virtue of being an android application in android devices and is app-specific, learn more about it [here](https://developer.android.com/training/data-storage/app-specific)) and use it for storing state. Also we'll need to refactor the FFI Layer as well after the changes being made in application's package name.
## **Describe alternatives you've considered**
<!-- A clear and concise description of any alternative solutions or features you've considered. -->
* Writing Custom impl for finding home(as xdg-rs does it incorrectly)
* modifying + extending the FFI Layer in accordance with application codebase.
* Adding [`android-support`](https://git.irde.st/we/irdest/-/tree/develop/utils/android-support) crate to our workspace to make it part of CI runs and track if it breaks at some point.
## **Additional context**
<!-- Add any other context or screenshots about the feature request here. -->
NAGSoC 20212021-07-02https://git.irde.st/we/irdest-corrupted/-/issues/15Remove Redundant Files from Android codebase2021-08-08T06:48:30ZGhost UserRemove Redundant Files from Android codebaseThe following files are present in the android application codebase that need not be tracked in version control
* [Application APK](https://git.irde.st/we/irdest/-/blob/develop/clients/android/app/debug/app-debug.apk), we needn't check i...The following files are present in the android application codebase that need not be tracked in version control
* [Application APK](https://git.irde.st/we/irdest/-/blob/develop/clients/android/app/debug/app-debug.apk), we needn't check it in to VC as it can always be downloaded from latest pipeline that ran on our develop branch
* [`Output json file`](https://git.irde.st/we/irdest/-/blob/develop/clients/android/app/debug/output.json), useless generated file that plays no role in VC.
* [`app/gradle/wrapper`](https://git.irde.st/we/irdest/-/tree/develop/clients/android/app/gradle/wrapper) dir, it needs to be present in root of the application codebase as it provides the gradle commands which we can execute on CI by being in root of application, not inside it.
* [`app/local.properties`](https://git.irde.st/we/irdest/-/blob/develop/clients/android/app/local.properties), personal config file of person who is working on the project, it contains path to android-sdk which definitely may differ for each person depending on their config, wrong path to sdk will not let the application build/sync properly.GSoC 2021https://git.irde.st/we/irdest-corrupted/-/issues/12Alexandria internals refactor2021-08-07T06:44:05ZKatharina FeyAlexandria internals refactorAs part of the NLnet grant to work on Irdest I allocated a milestone to get Alexandria to the `1.0` finish line. It's an ambitious target, seeing as the database is not yet [5 years old](https://twitter.com/sadisticsystems/status/942334...As part of the NLnet grant to work on Irdest I allocated a milestone to get Alexandria to the `1.0` finish line. It's an ambitious target, seeing as the database is not yet [5 years old](https://twitter.com/sadisticsystems/status/942334089578680320) (:wink:) but it will be important to be able to rely on storage in the future, and hopefully this will also make the `irdest-core/store` module obsolete!
Following is an outline of work that needs to happen. This issue will track development of these features.
## Alexandria: re-scope
As part of qaul.net Alexandria needed to handle many types of payloads. This is no longer the case! As such `blob` support needs to be removed. Furthermore, we will need to support large records, and possibly records where a key-value store approach is not ideal. Following is an outline of the **new alexandria scope**
* Encrypted data & metadata
* Identity concealing. Hide user IDs. Is this zero knowledge?
* Records based. Path based. Two types of records
* Key-value stores
* Table records with shared schemas
* Search tags. Encrypted tag cache. Per-user.
* Encrypted schemas.
* Insert data easily without manual diff creation
* Use diffs for transactions internally.
* Generational (epoch based) garbage collection
* Sync data dynamically.
* Break large records up into chunks that can be streamed from disk.
### Things that are stored
* Records! Chunk based.
* User sessions, with hidden user IDs. How?
* Per-user tag-cache
* Per-user private key store
* Per-user public key store (not encrypted - only user ID is unknown)
### The threat model
Hiding information from an adversarial user is impossible. Alexandria can not hide its own memory from the root user. Data MUST exist in un-encrypted form at some point or another.
Watching alexandria in memory WILL reveal information about data present in the database.
Protect data synced to disk. Confiscated devices, broken disk encryption, stolen files. Don't reveal user identity or information payloads to on-disk attacks. If the cops steal your phone, can they indict you?
## Internal overview
Following is a graph of data available via APIs, internal components and their on-disk counterparts.
![alexandria-overview](/uploads/efd08ca912588d6fba0a0dde8142690a/alexandria-overview.png)
## Outline
Following is an implementation outline
- [x] Investigate whether the `Encrypted` abstraction introduced in alexandria `v0.1` is still relevant
- [ ] Take a closer look at every component that exists in the code currently and map its inputs and outputs
- Where are modules being used?
- Are there unused modules in the library?
- [ ] Build a solid user session management module
- [ ] Allow user registration, login, logout, destruction
- [ ] Sync user sessions & keys to disk
- [ ] Tag & schema cache
- [ ] Encrypt caches with root user key
- [ ] Provide an API for other modules to query the caches
- [ ] Load information from disk that is not present in the cache (how?)
- [ ] Records store
- [ ] Investigate whether to split the store implementation between KV and tables or keep them in the same stare
- [ ] Create an API for manipulating tables & diffing KV records
- [ ] Create transactional deltas for the store
- [ ] Apply deltas atomically
Add and remove items from the above outline as needed.https://git.irde.st/we/irdest-corrupted/-/issues/9Reduce the amount of warnings2021-05-13T15:53:17ZKatharina FeyReduce the amount of warningsCurrently we have a lot of `unused code` warnings around the codebase. Some of them will go away naturally as we connect more pieces of the puzzle together, but others can be eliminated by either
1. Deleting code that is no longer need...Currently we have a lot of `unused code` warnings around the codebase. Some of them will go away naturally as we connect more pieces of the puzzle together, but others can be eliminated by either
1. Deleting code that is no longer needed
2. Making API endpoints `pub` that are currently only `pub(crate)` or similar
3. Gating testing-only API endpoints behind `#[cfg(feature = "testing")]` or similarhttps://git.irde.st/we/irdest-corrupted/-/issues/8Rename the project2021-08-03T09:03:41ZKatharina FeyRename the projectWe have lots of references to the old name in the repository. Most of them have been taken care of (in code anyway), but a lot of documentation still mentions `qaul`.
This is hard to fix automatically, especially because `qaul` and `ir...We have lots of references to the old name in the repository. Most of them have been taken care of (in code anyway), but a lot of documentation still mentions `qaul`.
This is hard to fix automatically, especially because `qaul` and `irdest` don't map perfectly onto each other gramatically. Thus we should do this process by hand. Any MRs to fix spelling, naming, etc are welcome. Please reference this issue in your MRs!https://git.irde.st/we/irdest-corrupted/-/issues/7Improve irdest-core storage API2021-05-28T14:59:05ZKatharina FeyImprove irdest-core storage APICurrently the `services` section of the irdest-core API has three main endpoints that services can use to store data.
- `save(UserAuth, Service, MetadataMap, TagSet)`
- `delete(UserAuth, Service, Key(String))`
- `query(UserAuth, Service...Currently the `services` section of the irdest-core API has three main endpoints that services can use to store data.
- `save(UserAuth, Service, MetadataMap, TagSet)`
- `delete(UserAuth, Service, Key(String))`
- `query(UserAuth, Service, TagSet)`
The `MetadataMap` type is defined as follows:
```rust
pub struct MetadataMap {
name: String,
map: BTreeMap<String, Vec<u8>>,
}
```
The problem with this approach is that diff-operations need to be handled by the service, and this logic can not be re-used by other services.
## The solution
* [x] Remove `save` function endpoint
* [x] Create an `insert` function endpoint with the following properties
* Each service has access to **one** store map (the `MetadataMap` may be re-usable)
* Function API takes `(UserAuth, Service, Key, Value)`
* `Key` is a `(String, String)` tuple, that acts as (Namespace, Key)
* `Value` is bound as `V: Serialize` to be used by the service as it sees fit
* [x] The `delete` and `query` endpoints can be re-used, albeit their function APIs need to be adjusted.
* `query` will now only return a single T, bound `T: DeserializeOwned`.
Some future considerations to make the general storage subsystem in irdest-core simpler is to approach the alexandria API differently. Notes towards this should be tracked in #4.https://git.irde.st/we/irdest-corrupted/-/issues/2Combine TCP and UDP driver implementations into `netmod-overlay`2021-05-12T23:28:43ZKatharina FeyCombine TCP and UDP driver implementations into `netmod-overlay`Currently there are two overlay `netmod` drivers: `netmod-tcp` and `netmod-udp`. Both are important when building a qaul network via existing networking infrastructure.
Because the UDP and TCP drivers share a lot of the same abstractio...Currently there are two overlay `netmod` drivers: `netmod-tcp` and `netmod-udp`. Both are important when building a qaul network via existing networking infrastructure.
Because the UDP and TCP drivers share a lot of the same abstractions (mapping `ratman` IDs to IP addresses), code logic is duplicate. Furthermore, it might not be clear to someone not versed in networking why they should use the UDP or TCP modules, and what advantages and disadvantages these two protocols offer.
To improve this experience, as well as the maintainability of the network drivers in the future, the plan is to deprecate `netmod-tcp` and `netmod-udp` and to introduce a new driver called `netmod-overlay`, which implements the same `ratman` IDs -> IP addresses mapping mechanism, while being able to use it for both TCP and UDP connections. Optionally, we should investigate using [QUIC] instead of TCP, as it can be overlayed on the UDP connections, and may reduce transmission overhead to improve overall network throughput.
[QUIC]: https://en.wikipedia.org/wiki/QUIC