Overview

This is the documentation of buenzlimarks (GitHub).

buenzlimarks is:

• a bookmark management application
• free and open-source software (license)
• made to be self-hosted

Its not-yet-implemented standout feature is PWA-based offline use.

In this book, you should find all the information you need, whatever buenzlimarks is to you. Please jump to the section most relevant to you:

1. User Manual (TODO)
2. Self-hosting buenzlimarks (TODO)
3. Contributing to buenzlimarks
4. The architecture of buenzlimarks (arc42)

Contributing

Here you will find information about contributing to buenzlimarks.

• Server development workflow tips
• Testing Policy

Server development workflow tips

This is a list of helpful tips and pointers to more resources related to developing the server.

Getting started

First of all, run just in the terminal to get a list of the available recipes. There should be one to run the server and one to fill your development database with seed data.

Note that the development database is in the directory dev/db. You can use the file system directly to check the contents of the database when needed.

If you encounter any terms you don't understand, check the glossary. If the term isn't in the glossary, ask about it so it can be added.

Making sense of the server directory

There are two main perspectives that are needed to understand the server. Both are documented well in the architecture documentation, find links to the specific sections below.

The first is the static building block view. It is concerned with how the code is structured into modules (folders), the purpose of those modules and how they relate to each other. The building block view is documented here.

The second important view is the dynamic runtime view. It is concerned with the sequence of events and lines of communication while the server is running. The most important "unit of execution" for a web server is the API call. A user makes an API call over the internet, meaning they ask the server to do something or provide some information. They wait for the server to respond in some way. The server handles such a request in a specific pattern. To understand this pattern, refer to the sequence diagram of an API call in the architecture documentation.

I want to add a feature... how do I do that?

In order to add or change a feature, you will most likely have to change something in relation to every step outlined in the sequence diagram of an API call.

If you change everything at once and the result isn't what you want, you're gonna have a bad time looking for the problem. Make sure you change only one thing at a time and verify that your change works at every step of the way. The best way to do this is to write a unit test for your change. The modules handlers and db already contain examples of unit tests. Before your changes can be merged, they need to be unit tested anyway. Doing that part first makes the rest of the work that much easier.

To test your changes with a proper API call, consider using the vscode extension Thunder Client. An example API call that should work is already stored in the repository. Make sure the server is running when you're making API calls.

Everything related to routing is managed closely by the axum framework. Without any knowledge of that, the routing can be hard to understand. Refer to the documentation of axum's Router.

When adding or changing a handler function, you may sometimes get cryptic errors. The router expects the handlers to conform to a generic type which can lead to bad error messages. axum provides a macro that can improve these errors. Simply add this line right above your handler function defintion: #[axum::debug_handler].

Testing Policy

1. When something's broken that could've been prevented by a test, we amend this testing policy document¹.
2. We don't do any testing.

Arc42 Table of Contents

1. Introduction and Goals
2. Architecture Constraints
3. System Scope and Context
4. Solution Strategy
5. Building Block View
6. Runtime View
7. Deployment View
8. TODO Cross-cutting Concepts
9. Architecture Decisions
10. TODO Quality Requirements
11. TODO Technical Risks
12. Glossary

Introduction and Goals

This document describes buenzlimarks, a bookmark management application. It is used to store and organise browser bookmarks and related things, such that they can be presented for fast and convenient access later.

The following goals have been established for this system:

• The system shall have all features of current competing products which are actually used by the target user base (family & friends of the developers).
• The system shall be highly available. Offline use is of equal priority to online use for UX considerations.
• Development of the system shall be an opportunity to gain experience with cutting edge technologies for the developers.

Requirements

Requirements are developed and documented as GitHub issues. They can be found here.

Quality Goals

The architecture of the system is designed to fulfill the following goals:

• Availability: The UX during offline use is of the highest possible quality.
• Efficiency: The self-hostable backend server is resource efficient to enable deployment on cheap hardware like a raspberry pi.
• Extensibility: The system is easily extensible to store information beyond simple website links as well as provide more UI customization capabilities.

Stakeholders

This is an overview of the stakeholders who

• should know the architecture
• have to be convinced of the architecture
• have to work with the architecture or with the code
• need the documentation of the architecture for their work
• have to come up with decisions about the system or its development

Architecture Constraints

Web technologies

The application must run within browsers, anything else would constitute bad UX. The architecture is thus constrained to fulfill its goals with the web technologies available in modern browsers. (e.g. HTML+CSS+JS+WASM, service worker capabilities, browser extension APIs)

Limited server resources

The backend server must be self-hostable where compute resources may be limited like a raspberry pi. It must therefore be relatively resource efficient, which constrains the choice of technology used for its development.

System Scope and Context

buenzlimarks is mostly a standalone web application. Users interact with it through their browser. buenzlimarks uses the browser's IndexedDB to cache user data locally. OAuth providers are used for lightweight user authentication.

Solution Strategy

Quality Goals

Technological Decisions

Web app

• capable of fulfilling our key quality goal of first-class offline support via a service worker
• the only pragmatic option, as a bookmark manager that doesn't run in the browser can't deliver a great user experience

Rust on the server

• zero-cost abstractions and performance on par with C and C++
• loved by many programmers
• a modern type system with null safety, algebraic data types, powerful generics, exception-free error handling and no inheritance
• our developers are experienced with it

Top-level / Architectural Design Patterns

• frontend: TODO
• backend: traditional layered architecture

Organizational Decisions

Implicit API specification

OpenAPI is a great tool to achieve strongly typed interactions with a well-defined API. However, the code generators needed to make an openAPI based workflow productive are usually lagging behind for cutting edge languages and frameworks. Because we want to use these, openAPI is much less appealing. Moreover, the development team is small and informal communication about API changes is sufficient.

Development process

The team organization is lean and informal, as there are only two developers and a product owner. Scheduled meetings in the spirit of a sprint review, retrospective and planning combined are held every four weeks. Requirements are developed in a GitHub project, where individual developers may organize their tasks as well. Additional communication / meetings may always be initiated by any team member. The junior backend developer is responsible themself to seek guidance by the lead developer when needed.

Building Block View

Level 1 - Containers

buenzlimarks is mostly a standalone application. Users interact with it through the web app built with React. The web app installs itself with a service worker for offline use upon first visit. Additionally, users may manually install a browser extension to interact with. Both the web app and the browser extension access the browser's indexedDB for locally cached data. Syncronization between different devices is achieved by talking to the Rust / Axum backend via HTTP and optionally websocket. OAuth providers are used for lightweight user authentication.

The following container diagram shows a decomposition of the buenzlimarks software system into its main containers. These containers are individually deployable and interact with each other only through protocols like HTTP & websocket. For convenient deployment, the web app may be bundled with and served by the backend.

Web app

The main GUI for interacting with buenzlimarks. Installs itself via a service worker for offline use upon first visit. Based on HTML, CSS, TypeScript, React, Tailwind. Accesses user data locally via indexedDB, remotely via the REST API and may listen to remotely updated data via websocket.

Backend server

Stores user data for synchronization across multiple devices. It's written in Rust with the Axum framework and exposes a REST API.

Browser extension

Offers a subset of the features of the web app for convenient access anywhere in the browser. Accesses user data locally via indexedDB, remotely via the REST API.

Interfaces

The web app and browser extension have a somewhat implicit interface. They need to agree on the structure of the data saved locally in indexedDB. The backend server is mainly interacted with over its REST API which is not explicitly documented, e.g. with openAPI. In addition, the web app may open a websocket connection with the backend server, which will then notify the web app about changed data, e.g. by another device of the same user.

Level 2 - Components

Web app components

TODO document interal architecture of system components as it emerges. Prefer relevance over completeness. Specify important, surprising, risky, complex or volatile building blocks.

Backend server components

The server is divided in three layers or modules:

• db: The persistence layer or database, everything related to storing data to and fetching data from disk.
• handlers: A collection of functions, each responsible for "handling" a different kind of API request.
• models: The domain layer, including the data model and, if applicable, any associated business logic.

It deviates from a traditional layered architecture in that the handlers call the persistence layer directly. The domain layer is only called in specific cases where business logic is actually involved. This simplifies the majority of cases and avoids a lot of boilerplate, but may turn out to be more error prone if more business logic than expected is needed.

db (persistence layer)

Responsible for fetching and storing data in the database, be that a filesystem or a relational database. Moreover, the persistence layer is responsible for ensuring relational validity of the data. For example, if the database contains a bookmark referencing a widget with a given ID, a widget with such an ID must actually exist. Lastly, the persistence layer may expose an API for simple data pre-processing including sorting and filtering. This is good practice as relational databases are able to perform these operations very efficiently.

handlers

The handlers are bundled in hierarchical routers. The routers determine which handler is responsible for a request with a given route and method. For example, the top-level router may contain one nested router for each domain entity (bookmark, widget, etc.) respectively. These nested routers in turn may specify one handler for each of the four CRUD operations (create, read, update, delete).

Browser Extension

TODO document interal architecture of system components as it emerges. Prefer relevance over completeness. Specify important, surprising, risky, complex or volatile building blocks.

Level 3 - Code

Particularly complicated and or architecturally important code elements may be documented here.

Runtime View

API request

Deployment View

The web app is compiled and bundled using vite, the output goes into app/dist. The backend server is run with cargo run --release. It serves the compiled web app on top of its own REST api on port 4000. On the server where the software is deployed, nginx or caddy may be used to handle SSL and forward the plain HTTP requests to the backend server. Let's Encrypt and certbot may be used to acquire SSL certificates.

Dockerization is something to consider for a more streamlined deployment experience in the future.

Architecture Decisions

1. SolidJS & TailwindCSS Single Page Application
2. Rust & axum backend
3. Persistence with SeaORM - SUPERCEDED
4. Simple Filesystem Database
5. React Single Page Application - SUPERCEDED
6. Leptos Single Page Application

SolidJS & TailwindCSS Single Page Application

Context

Writing a web app with JavaScript basically requires the use of a UI framework, as vanilla JavaScript is incredibly unproductive and leads to unmaintainable UI code. The lead developer has experience with React, which enables faster development. SolidJS is similar to React from a development perspective, transitioning to it should have a gentle learning curve. Solid is much newer more "cutting-edge", but also less mature in many ways. In comparison to React, Solid:

• has a better developer experience
• requires less boilerplate
• sports better performance
• is particularly lacking in component libraries, which can be mitigated by choosing a framework-agnostic styling system like TailwindCSS.

Framework popularity may be important for projects looking to hire new people, which is not the case for buenzlimarks. The lead developer has neither experience with, nor interest in, other frameworks than React and Solid.

Tailwind generally requires more work for, but gives more control over, the styling of the web app compared to more "batteries-included" component libraries like Material-UI. Component libraries on top of Tailwind do exist, one example being daisyUI.

Decision

We will write the web app with Solid and Tailwind.

Status

Partially superceded by React Single Page Application.

Consequences

Writing the web app will be a learning experience. However, it will also generally be more work and take longer. The resulting code is expected to be more maintainable. Styling customization as a user-facing feature can be provided easily.

Experience report

SolidJS has a few developer experience pitfalls just like React does. For example, object destructuring breaks reactivity in some cases. The argument that SolidJS would provide a better developer exerience than React is much weaker in hindsight. The smaller, less mature ecosystem has been much more noticeable.

Rust & axum backend

Context

The backend of a web app can be written with virtually any language, contrary to the frontend. buenzlimarks' backend must be resource efficient for deployment on cheap hardware.

This basically eliminates Python and strongly disfavors languages whose ecosystems are characterized by "enterprisey", resource-hungry frameworks like Java and C#. JavaScript is quite performant with V8 and TypeScript allows writing quite robust and maintainable software. C and C++ are performance beasts, but suffer from decades old lanaguage foot guns.

Go is highly performant, but its language design is outdated. It has no algebraic data types or null safety, weak generics and modularity. It even introduces some entirely new foot guns, like tedious to circumvent and error-prone zero-initialization of structs.

Rust has no garbage collection like C/C++ and matches them in performance. It avoids all their foot guns and provides important modern language features like algebraic data types, null safety and strong generics. It goes very far in preventing common developer mistakes at compile time. For example, it guarantees thread safety.

When writing a web server, it is common to use a library / framework to provide abstractions over the HTTP protocol. A language with a strong ecosystem is necessary to reduce the amount of manual work required.

JavaScript has many such libraries, including ones specifically designed for the most popular UI frameworks. The same language on the server and the client enables a very nice developer experience, trpc being a great example of that.

Rust has a lively, almost volatile webserver ecosystem. Actix Web and axum seem to be the current community-favorites. Axum is quite new, made by the developers of tokio (the most popular async runtime) and is designed for simplicity and modularity.

All current developers are experienced with Rust, with is not the case for JavaScript.

Decision

We will write the web app with Rust and axum.

Status

Accepted.

Consequences

The performance of the backend server will be perfect for running on constrained hardware. Gluing together the modular pieces will require more work and architecural considerations. The resulting code is expected to be highly maintainable and robust.

Experience report

Persistence with SeaORM - SUPERSEDED

Context

An ORM is used to avoid the tedium of manually writing SQL queries. Diesel seems to be the "default" choice in the Rust community and version 2.0 is about to be released. SeaORM is another, younger alternative worthy of consideration. It hasn't yet reached 1.0, though. Both ORMs support PostgreSQL, MySQL and SQLite. SQLite will probably suffice for our purposes and a later migration to Postgres should be easy enough.

Diesel isn't well suited for async!

Diesel seems appealing, being the more "mature" option as well as having stronger compile time checks. However, SeaORM has quite lovely documentation, especially for newcomers to the concept of an ORM, while Diesel is lacking in that area.

Our developer team has little to no experience with ORMs.

Decision

We will write our persistence layer with SeaORM.

Status

Superseded by Simple Filesystem Database.

Consequences

Database access is abstracted away, saving our developers from manual labor. Future migrations from and to database implementations are easy.

Experience report

An ORM feels quite heavy-weight, even overkill, for our simple web app. Although the documentation is great, the concept of an ORM, combined with relational databases, increase the learning curve for our junior developer significantly.

Simple Filesystem Database

Context

Relational databases and ORMs are difficult concepts for absolute newcomers, whereas the filesystem is familiar to any developer.

Reading and writing plain files is much less efficient than a relational database. This performance drop likely won't be noticeable for users, as our traffic will be very low.

To make a future transistion easier, a filesystem database can be written against an opaque interface. This would enable an implementation of the persistance layer based on a relational database to be swapped in, while barely affecting the code using the database.

Decision

We will switch our persistence layer from SeaORM to the filesystem.

Status

Accepted.

Consequences

Onboarding of inexperienced junior developers will be much easier. Performance of the backend will be much lower, though still not noticeably bad.

Experience report

React Single Page Application

Context

In order to speed up development, component libraries are often used. These are not useable in our context, because we need full control over styling and have chosen Tailwind for this reason. A more recent alternative to component libraries are headless UI libraries, which provide the logic and accessibility of UI components without any styling. Radix is one of the leading headless UI libraries and it only supports React.

The supposedly better developer experience of SolidJS has been less advantageous than anticipated, as described in this experience report.

Solid and React remain very similar frameworks in terms of the code we write, migrating back and forth requires relatively little effort.

Decision

We will switch our client-side UI framework from SolidJS to React.

Status

Superceded by React Single Page Application.

Consequences

Finding high-quality libraries we need will be easier (e.g. Radix UI). Performance will be worse, though likely unnoticeable.

Experience report

SolidJs or React, JavaScript is just annoying.

Leptos Single Page Application

Context

We recently discovered that the relatively young Rust frontend framework Leptos is very mature, fully featured and well documented. Since the reactivity model is very similar to React / SolidJs and Tailwind is supported, a migration doesn't entail too much work. The advantage of such a migration is a complete Rust stack. The downside is the loss of several libraries only found in the JavaScript world, notably tanstack-query and any headless UI libraries.

Decision

We will switch our client-side UI framework from React to Leptos.

Status

Accepted.

Consequences

Finding high-quality libraries we need will be easier more difficult if not impossible. Bundle size will be larger, due to compilation to WebAssembly. The development environment and workflow will be much more streamlined. There will be drastically fewer useless configuration files. Our junior Rust-developer will be able to contribute to the frontend as well.

Experience report

Glossary

• Domain specific terms
• Technical terms

Domain specific terms

Technical terms