Posts

Posted on

The Mediocre Machine

In my first article, I looked at Generative AI delivering the Plausible Looking Answer. I did not go into the mechanisms of how Large-Language-Models work. This time I’d like to layout how understanding their technology also helps understanding their short-comings and reduces anthropomorphizing them over and over. Let’s try and have a look into the machine.

Two phases, two layers

To talk effectively about it, we have to look at two phases of working with the Large-Language-Model.

The first is the training phase in which LLM providers feed as much data as they could get their hands on into the model. The training takes months, a-hell-of-a-lot of (copyrighted) data, energy and cooling water. Once training finished, the model is frozen and made available to use by in or by other services.

Once the model is running, you have the first layer of your AI-service – its foundation, hence “foundational AI model”.

The model itself is frozen and state-less. You send it an request and it will generate an answer. It will not learn on or store your input, your prompt. It will not “remember” what you did a minute ago. To achieve something like short-term-memory, we need to build something on top in the application layer. This layer orchestrates between user inputs, access to additional resources, prompting the LLM. receiving and displaying its response. Much of what happens in an AI-services is not in the actual LLM, but happens in the application layer. We will go deeper here at a different time.

The orchestration also makes the model start generating.

Wakey wakey

The lifecycle of an AI request goes through the same three steps continuous steps: the LLM is state-less program running on a server. The server idles waiting for a request, upon receiving a prompt, the server asks the LLM to generate a response – streaming it piece by piece. Upon finish, the server goes back into idle – the LLM not performing any action. Why is it streamed piece by piece? In an attempt to make waiting times of slower models feel shorter.

If you don’t prompt it, the server sits on stand-by. Without a prompt, the LLM is not neither “thinking” nor “resting”. It will not have a spontaneous idea and “wake up”. The LLM behaves like any other server, waiting to be called. The prompt-response-pattern is at the core of every AI service.

Averaging the human knowledge

The process of creating and training a Large-Language-Model is complex and my explanations will be simplified to get the gist.

Training can be [split into more steps](https://magazine.sebastianraschka.com/p/new-llm-pre-training-and-post-training) (Pre-training, Post-Training, Finetuning, etc) but for us the *Pre-Training* is the most important.

What is this Pre-training? It’s the GPT in ChatGPT (Generative Pretrained Transformer). Today’s LLMs are trained over months on all text, books, websites, movie scripts and movies. These have been broken down into Tokens. Instead of “Daniel” or each letter “D”, “A”, “N”, “I”, “E”, “L”, my name has been tokenized to “Dan” & “iel” and during pre-training the algorithm found, that after the token combination “Dan” and “iel” the tokens “Rad”, “cli” & “ff” are slightly more likely than “Sen” & “ff”. The same, but technically more complex happens when building models for audio or video, which are trained on audio, images and video as well.

Remember when I said in part 1 “No words have meanings to LLMs. To an LLM it’s all just symbols”? These tokens are the symbols. They hold no semantic meaning, they are the pieces for computation and for the statistical analysis.

Nothing the model was trained on is actually stored. It is one giant fuzzy database of parameters encoding the statistical relations between tokens and approximating the original data.

I just said approximation, because during training the original input has been compressed into these parameters with losses. It’s more like a vending machine: you select a product, the machine translates your input into the row and column of your choice, but you don’t know which exact article you will get. When selecting Cola, you may get regular Coke, or Diet Coke, or Cola Zero, or Pepsi. It all fits the original request, but something got lost on the way.

It’s not easy to imagine a multidimensional space of parameters describing an approximation of language and knowledge, so we won’t. We’ll stick with two dimension and look at an example.

Hallucinations is the mechanism

So you have your fully-trained LLM. When prompting it with a request. you basically ask the LLM “hey, here is this half of a thing, how would the second half look like?”. If you ask it a question, it will create an answer. If you give it an image and ask it to extend the edges, it will create a plausible looking scene, fitting with the original image.

This is fundamental to all LLMs, you pass the initial prompt and it responds with a plausible extension. What’s happening is that your prompt basically draws a path through the model’s billions of parameters and based on this path, it begins generating a response, that itself can go down many different paths.

By changing your prompt, you implicitly describe which parameters get activated in your response. If you ask “Which cars are red?” it may draw a very straight-forward path between the statistics of the word “which” and the word “car”, with a detour through “English language” and may return a response like “Fire engines are commonly of red color.”. Change the prompt to “You are a sports commentator, what is the red car on the circuit of Monza?” the path will also lead though the topics “sports commentating”, “motorsports”, “Monza circuit” and you get a high likely how that “red car” gets related to “Ferrari” in the resulting response “Red cars on the race track of Monza are often Ferraris.”.

Based on this behavior many prompting-strategies have been developed, where providing very specific context can direct the LLM through topics that vastly improve the quality of the response. However no matter the prompting strategy, the LLM will always generate the most plausible path through its data. The generated answer is always an hallucination – just sometimes more correct than other times. It produces text without intent, without interpretation, without malice, without “understanding”. It’s only statistics. The machine is always guesses a text without understanding its meaning, it sees the tokens and attempts to guess an answer from statistics. And sometimes, its guess is right.

Personally, I prefer to avoid the term “hallucination” as it is a euphemism for LLMs producing text which does not stand fact-checking. It’s a machine doing what it is supposed to do, but it is sold by AI providers as more than it can fulfill.

Think harder!

Some LLMs have been extended to work as “Reasoning models”. These add another processing step when generating answers. This is called “thinking”, and can be thought of as an extension to Train-of-Thought. The model will first generate itself a new prompt, that is then being used to fulfill the task. By thus covering a wider context and laying a more detailed path through its data, the hope is to cover more ground and improve the final generated result. This comes at the price of lower response times as the model has to generate even more text, using more tokens and spending more energy.

When addressing these criticisms of wrong responses being hallucinated AI-fans often respond, that this will be fixed in the future and it will get better. The truth so far is is: It will not, as it’s not possible by design with the current technology. Hallucinations are inherit to the design and for the past two years AI providers worked hard to circumvent the most obvious ones on the application layer through moderation. They can’t fix every specific cases on training in the model. Models have mostly grown through data size and scaling, with lower return-of-investment, the bigger they get. Some people already ask, if they have become as good as they can get.

Summary

Whatever you ask your LLM to generate. It will always generate its response from its training data, its corpus. It will recombine, but is it creative? It will not remember, have meaning or understanding. In training any creativity and heart will be averaged out into a statistically plausible response. It will not have creative new “thoughts”. It will walk paths many people before already walked on. It will respond to the next person prompting a similar question with a similar result.

Recombination can bring new ideas and creativity is often a recombination of familiar elements in a new context. To achieve this, you need an understanding of its meaning. Good prompting can lead the model to be a helpful tool, but recognizing creative ideas requires judgement by a human being in front of the keyboard. LLMs produce language, not content. This is where slop is born. I’m reminded of a quote by Kevin Sands, “It is never the tool that decides. It’s the hands-and the heart-of the one who wields it.”.

As such AI can serve as a multiplicator. If you are creative, AI can be one more tool in your toolbox to achieve cool things. If you don’t bring the initial skills, it will make your work look like everyone else’s. If you feed your text into a generative AI to have it reworked, you are literally asking it to remove your personal touch and make it more average and less yours.

Using AI does not make you stand-out, it makes you blend in with everyone else using it.

Posted on

The niche

For a while i’ve been looking for a niche, I have yet to find.
If you are a software development company, your product is expensive given the people power and complexity involved. So to have your work financed you have to go in one of two directions:

Either you find one big customer who pays one large amount for your work or you find maaaany customers to pay a small amount.

“One large amount customer” usually means one business customer with a strong dependency coupling, which may be good if the product relationship/product is decent, or bad, if it sucks. Either way you have a co-dependency.

“Many small amount customers” means software needs to scale to millions of users or needs a very special nische. To reach this number, given the initial costs, you need an investor and again you have a strong dependency.

A special role are agencies and software manufacturies, who don’t have an own product and just sell their services.

What I’m missing are examples, of companies who develop their own service or software, maintain a small team size, are bootstrapped and cash positive and have a product, that does neither require millions or users, nor is totally dependent on one sponsor. A product without strong growth pressure. A product that does not need to submit to rot economy.

Posted on

A Plausible Looking Answer

Generative AI based on Large-Language-Model (LLM) is here, but most people have a vague understanding on what this black box is – how it works, what to expect. This is fine, you don’t have to understand how a solar panel works for getting electricity. However it is important to have an idea of the technology with certain expectation. Generative AI seems like magic when in reality it is all statistics. It looks like human thinking, sensing and feeling, when it is playing pretend. Explaining how LLMs work technically is interesting and also tough, but you can still find some mental images that help to set expectations of what this technology can do and how it operates.

My fallible friend

Your generative AI is like an eager friend. You can ask them any question. They will always have an answer. They will rarely say, “I don’t know” and will bullshit or even gaslight you into thinking, they do have the answer. Oftentimes they are right, but being right yesterday is not indication they will be correct today. Whether they are correct or wrong, or whatever, you only know once you check or question their response. Until it is checked, the response is only plausible.

Sentence completion turned to 11

We all have used the keyboards on our Smartphones. Above the virtual keys, you get a word prediction, where the system tries to anticipate your next word, so you don’t have to type it out. This is partially trained by your typing and reflects common word combinations you may have used. For years, on social media people played with predictive texts. You’d start a sentence and have the word prediction on your phone keyboard complete it. Generative AI and Large-Language-Models just like this, but on steroids.
The interaction is always a call & response. You pass a prompt and the LLM will auto-complete your prompt with a plausible looking response.

Plausible looking responses

Much discussion is about LLM responses being correct or false, but these are words, that have no meaning to LLMs. No words have meanings to LLMS. To the LLM it’s just symbols. LLMS have no thinking, no sensing, no concept of reality, of right or wrong, of fiction and non-fiction. LLMs are statistics trained on the internet with all biases, short-comings, copyright-infringements, and grammatical mistakes, there are. The more context you provide, the better its calculation may pass an answer that not only looks plausible, but is actually correct. You can ask an LLM, what the name of the first city on the moon is. If it says “New Berlin”, it gave you the answer from Star Trek. Technically not wrong, not totally correct either.

Similar when you ask for sources. The LLM will always try to create a plausible looking answer. In the answer, you’d expect a list of book titles. Having book titles makes it look more plausible than saying “I don’t know any books of that topic.” Thus it invents book titles, to make a more plausible looking answer.

On a limited scale, you can influence this with your prompt and the AI providers try ensure good looking answers, but this is a fundamental characteristic of the technology. It will generate plausible looking answers.

Hallucinations

When marketing departments companies pushing AI found that too many would generate garbage or fudge details, the term “hallucination” was coined. “Ah, the model is just hallucinating, it’ll improve in the next version”

Hallucination is not the failure of Generative AI, it’s the mechanism by how it works. It’s the statistical recombination that creates a plausible looking answer, given the initial prompt.

It looks like thinking

“But look the train of thought, it’s thinking, it’s reasoning!”

Train of thought is the method of asking the LLM to write down “what are the steps you would do to solve X”. Again, the result will be a plausible looking sequence of steps, but it’s not “thinking” and it’s not the steps the model will actually perform. It’s a generated answer describing the steps. Similarly, when you posed a question to an LLM and ask it “how did you do this”? It will generate a plausible looking answer, describing what a thinking process may have looked liked, but it didn’t think this process, neither has it memory. Apple recently published a paper going into more detail. If you ask a model “how do you feel, what do you want”, it will either respond with a plausible looking answer about being self-conscious and may sound very human doing that, but it’s still only a plausible looking answer. Many AI providers also tweaked their models, to respond to these kind of questions with template answers as too many – even experts – fell into the whole of believing to have found sentient AI. This is Pareidolia.

Doing Chain of Thought when prompting LLMs, it did show improved results by the AI’s response. What it does do is asking the LLM to cast a slightly wider net in it’s training data, thus getting a bit more context from the prompt, extending the level of detail to a degree where the generated response has more context based on fetched trained data and a higher chance of being correct.

How this breaks down can be nicely seen when asking math questions, or counting letters in a word. It doesn’t think, it generates plausible text. “How many letters are in strawberry?” A structurally plausible answer is “There are 5 r in strawberry” – there is no model of a letter, there is no model of a word or of the process of counting. It just knows, that in its billions of training data, statistics show, that a plausible answer looks like this. Applying train of thought, will have the LLM describe how letters are counted, but since it does not actually perform the steps it describes, it may still fail.

The strawberry example has been around long enough, that newer models were trained on articles describing this behavior. They may now return the correct number for strawberry, but now ask for blueberry and they may fail again. Similarly, when you ask to “generate a sentence of exactly 25 words” or “count the elements in this list”.

Summarize all the things

A common generative AI use case often presented are summaries. “Have this e-mail summarized for you!”. Unfortunately it doesn’t summarize. When you ask LLM for a summary, or a transcript, you will have one of two situations.

If the document you want to get summarized is about a topic, that is well documented and included in the pre-trained corpus of your model, when generating the summary it will draw heavily from this trained data and add “outside” knowledge to the document. It may add points not included in the original document, but which may be “plausible” to add. It’s like asking an intern to summarize a document and they will give you the Wikipedia summary.

If the document is on a topic not reflected in the corpus of your model, it can not draw from the training data, so it has to work with what you have given it. In this case, it will analyse the source document and make a statistical guess on what is imported in the document. This would be your intern counting how often the word “opportunity” was mentioned in the document and thus only mention the opportunity, while leaving out the risks because the word “risk” was only mentioned once. Can’t be that important, right?

And of course, a model may deliver a good summary, and still completely miss the point.

“Human-level” AI

To wrap it up, LLMs generate plausible looking answers by being a glorified word completion. It may be correct, but the more important the answer is, the more you need to critically review it. Humans are bad at review. I didn’t even touch on inherit biases and the devastating environmental footprint of AI.

Your AI friend can help you, or it can fail. Depending on the quality of the model, it can fail hard and invent everything in the quest for the most plausible answer. I know people talking about “human-level AI” to address this. AI is not infallible, it’s only “human-level”. I gave my sister the same explanation, on how LLM don’t do summaries. Her response was “still better than my colleagues”, mine “Get better colleagues”.

Posted on

Berlin building a Berg

Über die Jahre, die ich Berlin meine Heimatstadt nennen konnte, entdeckte ich ein Motiv, dass in der Geschichte der Stadt immer wieder aufkam. Eine stille Sehnsucht, die sich in einem regelmäßig wiederkehrenden Wunsch äußert. Dem Wunsch nach einem richtigen Berg.

2007 lernte ich im Dokumentarfilm “Die Pyramiden vom Treptower Park” erstmals von der einstigen Idee in der Stadt einen künstlichen Berg zu errichten. 1896 fand auf dem Gelände des Treptower Parks die Berliner Gewerbeausstellung statt, die ein deutscher Gegenentwurf zur Weltausstellung in Paris sein sollte, aber ziemlich verregnete. Die Gehwege des Treptower Parks lassen sich bis heute auf die einstiegen Ausstellungen der Gewerbeaustellung zurückführen. Ein Berg entstand nicht, dafür Pyramiden umgeben von viel kolonialem Rassismus.

Ein Jahr später wurde 2008 in Berlin der Flughafen Tempelhof geschlossen und das Tempelhofer Feld als neue Park eröffnet. In den Jahren um die Eröffnung gab es mehrere Ideen-Wettbewerbe über die Gestaltung des Feldes. Neben konventionellen Entwürfen, waren darunter auch eine Segel-Marina und … The Berg.

The Berg war der Vorschlag eines 1200m hohen Berges, aufgeschüttet auf der Fläche des 5km umfassenden Feldes. Eine Seilbahn mit Gondeln in Form von Rosinenbombern würde Skifahrer auf Skipisten bringen. Bergziegen säumen seine Hänge. Am Tempelhofer Bergstübl kann man für Almdudler, Spritzer und Aprés Ski einkehren. Die Illustrationen stehen bis heute online. Der Vorschlag geht mit einer Selbstironie heran un dem Wissen, dass Vorschlag wohl nicht realisiert werden wird, aber in unserem Herzen würde The Berg immer dort sein.

Kaum 10 Jahre später erwachte die Idee eines künstlichen Berges 2019 für zum 40. Geburtstag des Bezirk Marzahn-Hellersdorf zu neuem Leben. Das Künstlerkollektiv Plastique Fantastique errichtete am 7. September 2019 einen kunstvollen und künstlich Vulkan auf den Ahrensfelder Bergen, der für diesen Tag die höchste Erhebung Marzahn wurde.

Doch auch woanders kommt die Idee einen Berg zu errichten immer wieder auf. So auch in den Niederlanden von Thijs Zonneveld, mit dem Vorschlag einen 2km hohen Berg im sonst flachen Land zu errichten.
Italien hat indes die Idee bereits vor 1800 Jahren in die Tat umgesetzt.
Monte Testaccio ist ein 50m hoher Berg in Rom – aufgehäuft über Jahrhunderte aus antiken Tonscherben. Einst die größte Müllhalde des Antiken Roms beherbergt er heute unterirdische Restaurants.

Doch zurück nach Berlin. Was beim Enthusiasmus über das Errichten neuer Berge oft vergessen wird, ist dass Berlin dies bereits längst gemacht hat.

22 Jahre lang brachten täglich 800 Lastzüge bis zu 7000 m³ Schutt in den Grunewald. Bis 1972 entstand aus 26 Millionen Kubikmeter Schutt der Teufelsberg. Dies benötigte zuvor den gewaltvollen Abriss von 15.000 Gebäuden, einen Regierungsumsturz und eine wohlgesetzte Kugel am 30. April 1945. Ein Drittel des Bauschutt aus der Folge des 2. Weltkrieg endete im Teufelsberg. Weiterer Schutt schufen den Insulaner, den Mont Klamott und weitere Monte Scherbelinos in ganz Deutschland.
Der Teufelsberg 120m ist heute mit die zweithöchste Erhöhung des Gebiets von Berlin. Höher als die natürlichen Müggelberge im Südosten, etwas niedriger als die Arkenberge in Pankow im Norden. In 1972 eingeweiht machte er Westberlin ein neues Ski-Gebiet mit Sprungschanze und Schlepplift zugänglich. Der Lift wurde ein Jahr später wieder abgebaut.

Und so schließt sich der Kreis.
In Berlin sieht sich in seiner Sehnsucht nach einem Berg, dem Vergessen konfrontiert, was der Stadt bereits Berge gebracht hat.
Ich wünsch mir keine neuen Berge.
Wer Nazis wählt, ist ein Nazis. Wählt keine Nazis.

Posted on

Selbst-Organisation meiner Masterarbeit

Ich habe kürzlich meine Masterarbeit beendet. …

Das ist gelogen, das ist 11 Jahre her, aber damals begann ich diesen Text. Nach Abschluss meiner Arbeit wollte ich meine Prozesse dokumentieren, die mir während der Zeit Struktur und Hilfe gegeben haben. Dann kam Leben dazwischen und ich schrieb es nicht zu Ende. Bis heute.
Manches mag inzwischen aus der Zeit gefallen sein, aber so kommt das nunmal. Ich habe kürzlich meine ADHS-Diagnose erhalten und seitdem wird mir noch bewusster, wieviel ich in meinem Leben ADHS mit Struktur kompensiert habe. Die Masterarbeit ist hier ein Paradebeispiel. Also, wir spulen zurück.

Continue reading “Selbst-Organisation meiner Masterarbeit”
Posted on

Solving Tailscale DNS issues

As I mentioned the other day, I fell in love using Tailscale for making my local private network accessible remotely. I’m also using this in my company, but with one colleague, had an issue, I didn’t find any documented solution online. So here is mine.

I have a Tailscale network with multiple self-hosted services running in Docker and made available with Tailscale. If you have an account and your account is invited to the network, you can access them. This worked for 2 of 3 colleagues. The third had the Tailscale client running on their Windows, it showed up as active in the Tailscale admin console and the list of machines. It looks like it runs perfectly, but when you try access the the service within the network, it fails in Firefox with the error PR_END_OF_FILE_ERROR or SSL_ERROR_INTERNAL_ERROR_ALERT. In other browsers the error would just show as a connection error.

If you put the service in a public Tailscale tunnel, access is possible.
if you go on their machine, open Powershell and call tailscale status it would show all status just fine. Calling tailscale ping <service-name> shows success ping to the service.

I tested Windows defender and firewall settings, but could not find anything that could explain the issues.

Calling tailscale dns status provides on the machine provides an overview of the dns options Tailscale is using. Here it showed this:

=== 'Use Tailscale DNS' status ===

Tailscale DNS: disabled.

Tailscale is configured to handle DNS queries on this device.
Run 'tailscale set --accept-dns=false' to revert to your system default DNS resolver.

So I switched this on:

tailscale set --accept-dns=true

Et voila, calling the service work again!

So now I’m in this half-satisfactory space of having a solution, but not knowing the root cause. My guess is that this colleague hasn’t updated Tailscale since them installed it initially in October and when they installed the latest version, it did not install the latest default configurations properly.

My expectation would be that turn Tailscale DNS off again, would lead to the same issue, but it didn’t.

tailscale set --accept-dns=false

So something is weird, is resetted when activated DNS and stays good when deactivating it again.
Well it works, for now and I hope it’s a permanent solution.

Posted on

Disney gib mir Geld für mein “Inside Out 3”

Es gibt Filme, die holen einen im richtigen Moment ab. Inside Out 1 und 2 waren so Filme für mich.

Inside Out kam während eines “messy breakups”, einer Zeit, die von vielen mentalen Tiefs geprägt war und in der ich mich erstmals mit dem Thema Depression beschäftigte.

In Inside Out 2 erreicht Riley die Pubertät und mit Anxiety übernimmt ein energetisches Chaos das Steuer. Meine Anxieties waren mir lange nicht bewusst. Erst in den letzten Jahren lernte ich diese zu erkennen und einzuhegen.

Nachdem ich aus Inside Out 2 kam, ging ich durch das nächtliche Wien und machte mir Gedanken über die Filme.
In Beiden besteht ein eine gewisse Dissonanz wie “groß” das Ereignis in der realen Welt und wie heftig und episch die Reaktionen in Rileys Kopf sind. Wenn ein Umzug, zur gezeigten Verstimmung führt und ein schlechtes Wochenende zur Kapitulation vor Anxiety, ist die Wahrscheinlichkeit hoch, dass diese Episoden keine Einzelfälle sind und Riley öfter mit Mental-Health-Themen zu tun haben wird.
Mangelnde Regulation von Emotionen ist ein Symptom von ADHS. Wie cool wäre es, wenn Riley im nächsten Film ihre Diagnose als neurodivers erhält!? Oder im nächsten Film eine neurodiverse Person neben Riley gestellt wird!?

So gingen mir einige Varianten durch den Kopf, wie ich mir von Inside Out 3 vorstellen könnte.

Eine Frau geht durch einen langen Flur. Man sieht ihr Gesicht nicht, aber der blonde Pony lässt keinen Zweifel, dass es sich um Riley handelt. Sie trägt eine Brille – Nahaufnahme ihrer Augen – sie zeigen leichte Krähenfüße und deuten auf ein Alter Anfang 30. Sie stoppt vor einer Tür. Sie fasst den Türgriff, aber hält inne. Sie atmet ein. Das Türschild zeigt “Psychotherapie Dr Erica”. Sie atmet aus und betritt das Zimmer. Ihre erste Therapie-Stunde. Ein Neustart: 20 Jahre depressive Episoden, Selbstzweifel, Rejection-Sensitivity – verkrustete Glaubenssätze warten auf Aufarbeitung und Lockerung.

Zoom in ihren Kopf – an der Konsole streiten sich Ennui, Embarrassment und Fear.
Sadness, knackt ihrer Finger: “Lasst uns beginnen!”

Posted on

Everchords is sunsetted

Back in 2018 I had a problem. I was writing song lyrics in Evernote, but displaying them for playing was annoying. I started work on Everchords a tiny platform allowing you to connect your Evernote account and display SongPro lyrics beautifully. Over time I added more convenience features for songwriting that helped me a lot to streamline my song writing process. This was a big help when producing Bettlektüre.

All this ended a year ago, when I stopped using Evernote. So it wasn’t part of my workflow anymore. I kept doing security updates, but these took more effort lately. I thought for a long time how to continue it, but ultimately, I don’t know. I’m using Obsidian.md now, which does not have similar APIs. While all data is fully accessible being “just markdown files”, there is no default pipeline to process and there are so many ways to build this pipeline, that it makes little sense to build and maintain a full platform around this. So I decided to take offline.

The itch it scratched is still there, as I write my notes in , I may want to have a better songwriting experience again. We will see in time how I will change my process, adopt a different tool set or build something again to cover this.

Posted on

Next steps in Self-Hosting (pt 2)

In part one I spoke about the start of my journey, which was driven by playing with my smart home playground on topics of energy saving. My home servers have been in place since 2018 – yet, one thing I could never fully solve was how to make it available outside of my home network. For people into DevOps there will be little new here, but I’m no DevOps guy, so I had new things to discover I’d like to share.

So my challenge was to make my home server available via the internet, without making it public.

I was toying with a DynDNS setup at one point, but I never felt comfortable with exposing the full device to the internet. I know my way around Linux, but server security can become a bigger beast. I also didn’t want to invest in a virtual private server setup for this.

This is where Tailscale was a small revelation to me.
Tailscale is a service provider for virtual private networks. Basically, you install client apps on your devices building a virtual network and making machines available even when they are physically apart.
For example, I’m traveling and by flipping a switch on my phone, I see my PCs, my home server, my second phone, and I can access them as if they were in the same room with me. Each device gets a fixed IP and URL in the network and becomes way easier to access than remembering IPs and ports. I could add the Tailscale client to my Home Assistant OS running Raspberry and can access it from everywhere, as long as the Tailscale client is running on my device.
This was straight forward.

Schema of a Tailscale network. You have your home network at … home. This runs different services and has different devices. The home network is made available in your tailscale network either by having clients on your devices, or by inviting other people and their tailscale clients. Anyone who is not explicitely invited has no access.

What took me some time to learn was integrating not only physical devices, but also virtual containers. What is a virtual container? Basically, systems like Docker or Podman simulate a very slim computer environment specifically set-up and optimized to run one particular software stack. For example I can set up a web blog with the client frontend, backend server and database – each running isolated in their own Docker containers – easy to scale. Here you can find more about the basics of Docker and how to get into it step by step.
I had my first contact with Docker in 2018, but never got around to fully embrace it until now, but grew to love docker-compose quickly. This allows you to create and find recipes to quickly set up full tool stacks easily and self-host your own software infrastructure.

So I was playing and eventually fell into the sweet hole of self-hostable apps, extending my personal toolchain – especially for services for which I never found a good replacement. Now I run my Commafeed RSS reader, a gitea for my most private git projects, Jellyfin as a small media server and netflix alternative, Invidous giving an alternative frontend to Youtube and finally Immich.

What is Immich? I wanted to get rid of Google Fotos. I used the service for a long time with much appreciation, but eventually wanted to have more control over my data on the eve of Google using its foto database for AI training, I wanted to withdraw my pictures and also have better organized backups.
The easiest alternative I can recommend is ente.io which also comes with more tools around data security and multifactor-authentication (MFA). This option is simple to switch to, but still your data is not in your own infrastructure.

Immich has very similar functionality to Google Fotos and quick to set-up with Docker.
You can import your existing fotos either as an external library (very straight forward and simple, but at the cost of small limitations around cleanup options) or by importing all pictures via the API by uploading through the the web interface, the Mobile API, command-line scripts or additional tools like Immich-Go. My photo library has been growing in my filesystem since 2002 and while it is generally well maintained, it slowly became messy, incomplete, metadata was lacking and suffered from duplications. These were all topics Immich and its extended toolchain helped to address.
That is, if the photographs are imported in the “right way”. For me it involved three attempts on my side as image duplications on upload, mistaken metadata and lack of testing my own cleanup scripts (xD) caused several broken attempts with Immich. The software is stable and good, but my ambition in cleaning my library was just too big. May the backups be thanked – these setbacks did not cause any data loss – just wasted time. On the third attempt it worked well.up. I picked Immich as one example, but there are plenty more tools available to self-host and especially play nice with Tailscale.

Immich runs on my NUC 11 in a Docker-Compose setup and thus was now available in my home network, but again I wanted it accessible when I’m away from home. This is where we come back to Tailscale.
Above we only added physical machines to Tailscale, but it is also possible to add virtual containers to the private network. Each container becomes available as its own machine which now becomes available to everyone in your network to access.

Everyone? Oh, yes, I forgot. You can share your private applications in two ways.
You can invite up to 3 people (in the personal free account) to your network and they can have access to all machines if you allow it. Alternative you can share access to single machines to people who already run their own Tailscale network. This way you and your peers are building their own private little part of the internet hosted on your own infrastructure. You can also define machines as exit nodes and thus create your own little distributed VPN. Invite trusted friends over the world and you can run traffic through each other’s exit nodes and now I can watch German Netflix and my friends in Germany can watch Austrian ORF.

Tailscale is providing a good bundle to start your DevOps networking journey. All it offers can be done in other ways with less reliance on a central service orchestrating this network as Tailscale does. However doing this this manually is difficult, complex with a high risk of mistakes that compromises data security – these are factors I’d like to entrust to experts. Tailscale also has a learning curve, but one that is much faster to handle to get into the basics.

Through Tailscale and Docker I have a new toolchain that runs at home, that I have a lot of control over including full ability and responsibility to backup. I picked Immich as one example, but there are plenty more tools available to self-host and especially play nice with Tailscale.

In my next article, I will look into another example, that grew out of Docker and Tailscale at work and that describes a self-hosted alternative approach to a Data Intelligence pipeline in the context of a non-technical company.

Posted on

My first steps in home automation (Pt1)

I have not been writing code for a year, but my current job is still about providing and choosing technical solutions based on requirements and context. While the company I work for has a strong direction towards GenAI and Microsoft solutions (who doesn’t…), in my private projects I look into the other direction of Small Web, Decentralisation, Self-Hosting and the Rot economy of Tech these days. This also proofed a good opportunity to dive deeper into Home Automation and Docker. It’s an ongoing process, but here I’d like to talk about a few pet projects I did in the last year.

It started back in early summer 2023, I got myself some Shelly smart plugs , installed [Home Assistant]() and began to measure my energy consumption. This ran on my mini-pc – a small Intel NUC 11 Performance – I had been using for a year for backups, but little else. The NUC was an upgrade from a RaspberryPi 3 running dietpi I used for the same purpose. For convenience sake I kept using dietpi also on the NUC 11.

By measuring energy consumption, replacing inefficient devices (ffs get rid of old light bulbs!) I was able to reduce my already low energy consumption by another 20%. Having this visually not only helped reducing total consumption, but also shifting consumption to times windows which are either cheaper by having a lower hourly price or by optimizing for lowest CO2 usage by consuming when most renewable energy is in the grid. Often this coincides with low prices, but not always.

Since working on energy projects professionally and switching to a dynamic energy tariff in February I have been looking more into energy topics. In between I tried to build a predictor that could guesstimate the energy prices for upcoming days based on weather and other factors, but eventually I realized, that this is fruitless as the prices is a negotiated market price based on consumption expectation across all of Europe. To say it simple: the price is not only caused by weather, it’s set to provoke and steer market reaction. I have not the data to predict that.

What is more helpful is experience. On average daily energy is cheapest during the week between 3-4am and 11-2pm. Avoid the 6-8pm evening peak. On the weekend, consume between 11-4pm – sometimes you are lucky and zero to negatives prices. By timing dishwasher and washing machine accordingly, 50% of my consumption falls into the cheapest times.
On a regular day, I have 2kWh consumption split with laundry, kettle and dishwasher each at about 1kWh per run being the most hungry devices. Another 1kWh falls to my Tech and I can see in my consumption, when I run the gaming PC with VR. Despite my below-average consumption, I have been dancing around getting a balcony solar-installation for almost a year. At 300€ these have become sooo cheap… story for another day.

I managed all smart home devices in Home Assistant. in the beginning, this ran on the NUC managed by dietpi. Eventually in Spring this year I reactivated the raspberry after all, to run Home Assistant OS on it instead. This offers some more customization and supervising options, necessary to install more custom plugins and integrations.
Add the Home Assistant App to this and you have a neat remote for your home.
I stopped tinkering here with Home Assistant at this point, although topics like MQTT data broadcasting are still on the list of things I’d like to checkout one day.

Home Assistant at this point was in my local network and only accessible at home. How to make it available remotely and what other infrastructure I started hosting at home, I’ll cover in part two.

Posted on

Henlo, I bims neurodivers

Ein Kessel, der ständig überhitzt, unter Druck steht und das erste Mal abkühlt.
Die Welt läuft 5% langsamer und erstmals in der selben Geschwindigkeit wie ich.
Eine neue Ruhe im Körper – die Anspannung ist weg.

Elvanse lag 8 Monate in meinem Schrank. Zwei Pillen sind einer lieben Freundin vom Laster gefallen für “wenn ich mal schauen will, was an ADHS dran ist”.
Seit dem schob ich es immer vor mir her, dieses Experiment zu beginnen.

Dann kam der März. Am Ende meines ersten Experiment-Tages, schilderte ich ihr meine Eindrücke und sie schrieb trocken “Willkommen bei den Neurodiversen”. Elvanse ist ein Amphetamin, eine Stimulanz – mit einem neurotypischen Gehirn, wäre ich an dem Tag aufgedreht durch die Gegend gesprungen. [1] Stattdessen war ich ruhig, entspannt und konnte das erste Mal seit langem drei Stunden konzentriert arbeiten. Alles war zugänglicher, eine komplett neue Körpererfahrung.
Damit war es klar. Ich habe ADHS.

Seitdem habe ich meine offizielle Diagnose und ich rede regelmäßig drüber. ADHS endlich bestätigt zu haben ist für mich auch ein Coming-Out. I come in pride.
Neu ist mir das Thema nicht, Freunde hatten schon lange die Vermutung und ich wusste ebenso lange, wie ich mich in dem Thema wiederfinde. Gleichzeitig fühlte ich mich wohl Schrödingers ADHSler zu sein und nicht klar als neurodivers oder neurotypisch eingeordnet zu sein. Nun ist diese Einordnung eindeutig.

Die Reaktionen in meinem Umfeld waren fast durchgehend in einem von zwei Lagern:
Das “No, shit sherlock”-Camp, dass als Erstes fragte, ob die Diagnose für mich überraschend sei.
Dem gegenüber steht das Lager “Rehblick” – Menschen, die mich mit großen Augen überrascht und verständnislos anschauen. Deren erste Frage einfach ist, wie es mir damit geht. Danke sehr gut.

Seit dem Sommer hab ich die offizielle Diagnose und bekomme meine eigenen Medis. Ich bin aufgewachsen als jemand, dem immer eingetrichtert wurde mit Medikamenten solange zu warten, wie es geht. Dies lege ich langsam ab. Für Menschen mit Sehschwäche ist eine Brille ein total normale Hilfe. Für Menschen mit einem anders verdrahtetem Gehirn, sind halt Medikamente die Sehhilfe. Ich hab lang genug gezeigt und bewiesen, wie weit ich ohne komme und zu welchem Preis.

Ein Thema, nach dem ich auch gefragt werde ist, ob ich mit Medikamenten nicht die “Fähigkeiten” verliere, die mich aus machen, bzw die ADHS mir gibt. Ich bin top organisiert, super Problemlöser und klasse in Analyse, Pattern-Matching und letztlich vielseitig und kreativ. Ja, den Gedanke, das zu verlieren hatte ich auch, jedoch hörte aber von allen Seiten, dass dies nicht passieren wird. War auch nicht so. All dies blieb Teil von mir, ich gewinne aber Fokus um es besser umsetzen und stärken zu können.

Trotzdem ist es natürlich nicht nur rosig. Ich freue mich über die Diagnose und die Möglichkeiten nun Erleichterung zu finden. Ich sehe auch, wo ADHS mir in den letzten 30 Jahren viele Steine in den gelegt hat. Manche konnte ich selbst von alleine navigieren – andere nicht. Ich bin stolz, was ich in der Zeit alles hinbekommen habe und es gibt trotzdem auch Momente der Trauer, was mir verwehrt blieb und, dass diese Diagnose Teil meines restlichen Lebens sein wird. Einige Dinge werden niemals leicht werden und ich weiß, wie ich manche Themen derzeit auch noch nach hinten schiebe, wo ich selbst eine neue Position entdecken muss, bzw meine Alte reflektieren werde.

Ich dachte vor 6 Monaten es würde jetzt ein ewig langer Side-Quest beginnen, bis ich Diagnose und die Verschreibung habe. Aber ich bin sehr glücklich, dass durch gute Vernetzung und die Hilfe toller Menschen, alles sehr schnell ging. (Therapeuth*Innen, hört auf die Selbstdiagnose eurer Neurodiversen.)
Darüber bin ich sehr dankbar und nun nach einem halben Jahr kann ich mir nicht mehr vorstellen, wieviel Lebensqualität ich ohne die Medikamente auf der Strecke ließ.

[1] Eine andere Freundin schrieb später, sie hätte ADHS herausgefunden als sie gerade feiern war, sich Amphetamine einwarf und wunderte, warum sie plötzlich so hochkonzentriert und fokussiert ist.

Posted on

n8n – “Invalid DateTime” error on time processing

Over the past week I have been playing with a self-hosted instance of n8n – a Zapier-like automation platform. I quickly ran into an issue, that took a while to debug and I’d like to share my solution here.

I run a pretty standard docker-compose setup.

services:
  n8n:
    image: n8nio/n8n:latest
    ports:
      - "5678:5678"
    environment:
      - GENERIC_TIMEZONE="Europe/Berlin"
      - DB_TYPE=postgresdb
      - DB_POSTGRESDB_HOST=n8n-db
      - DB_POSTGRESDB_PORT=5432
      - DB_POSTGRESDB_DATABASE=n8n
      - DB_POSTGRESDB_USER=n8n
      - DB_POSTGRESDB_PASSWORD=n8n
    volumes:
      - "./n8n-node/:/home/node/"
    depends_on:
      - n8n-db
      - warehouse-db
    restart: unless-stopped

[…]

I live in Timezone Europe/Berlin, so I added the timezone ENV-variable.
The container starts fine and I can create workflows.

Now the weird thing was that date/time processing was broken.

Whenever I would create date/time converting nodes or expressions like {{ $now }} or {{ new Datetime("2024-05-12") }} or {{ DateTime.fromISO("2022-12-24") }} it would always bring up an error message Result: [invalid DateTime]

N8n is using Luxon library for date/time processing and I didn’t get why all provided examples where failing. Several searches and debugging did not bring me closer to a solution. I made sure I was using the latest version of n8n.
When I came across this question, where somebody had a similar problem a year ago. This pointed towards the timezone.

I had my time zone configured as:

…
    - GENERIC_TIMEZONE="Europe/Berlin"
–

Which turns out is just wrong… writing it this way has the quotation marks as part of the string thus the timezone n8n was parsing wasn’t ‘Europe/Berlin’. but ‘”Europe/Berlin”‘.
Removing this faulty line, the date/time processing with Luxon worked like a charme!

…
    - GENERIC_TIMEZONE=Europe/Berlin
–

So getting Invalid DateTime means, luxon fails to parse the timezone and can’t return a valid result. You can try this also with unknown time zones like “Europe/Berlin123” – same error.
So fix your time zone config and the error will vanish!

Posted on

Meine Podcast-Empfehlungen

Seit über 15 Jahren gehören Podcasts zu meinem täglichen Medienkonsum. Da mich auch immer mal wieder die Frage erreicht, welche ich empfehlen kann, gibt es hier meine Liste geordnet nach verschiedenen Themenbereichen. Dabei sind nicht nur aktuelle Podcasts, sondern auch einige Abgeschlossene. Ich finde nämlich Podcasts schließen viel zu selten ab und ihr Ende wird oft viel zu lang hinausgezögert.

Continue reading “Meine Podcast-Empfehlungen”
Posted on

The Weekly Dependency Chart

In the last 5 years I as Product Manager, I often found myself disliking the common ways on how to express project status and deadlines. They’d either hide important complexity or are a mess to keep up to date. Over time have grown a visualizations technique and meeting format I call the Weekly Dependency Chart. This helped me get structure and overview in projects I inherited, that were complex, late or required reliable schedules. I applied this in several private and professional contexts successfully. It was originally inspired by the Puzzle Dependency Charts used by Ron Gilbert in his adventure games.

The idea is to have a quick high level overview of timelines, to-dos, blockers all lined up in a weekly calendar view. Each item connected to the items it’s dependent upon.
Over time this graph evolved into a meeting structure, you can use to keep the overview up to date while synchronizing the status with the people involved.

Continue reading “The Weekly Dependency Chart”
Posted on

Holzschemel

Platt drückte sich ihr Hintern auf die Sitzfläche des weißen Holzschemel. Sie merkte die Unebenheiten seiner Oberfläche. Die platzende weiße Farbe täuschte nicht über den abgenutzten Zustand des Holzes hinweg. Der kleine Spalt würde nach dem Aufstehen eine kleine Markierung auf ihrer Wade hinterlassen. Kein Schmerz, aber sie spürte es. Wie das Jucken, dass ein jeder Mensch hat, wer bei geschlossenen Augen in sich reinhört. Sie lächelte.

Sie rutschte etwas, betastete seine Beine. Ihr Blick wanderte ihrer Hand hinterher, wie sie an einem entlang stricht. Dieser sollte es sein. Er war perfekt. Sie nahm die Säge und setzte an.

Drinnen war nichts zu hören. Niemand hegte verdacht.

Posted on

Kebus – Bus Ticket Apps Near You

Im Dezember war ich eine Woche in der nähe von Neapel unterwegs. Mehrmals kam es, dass ich aus dem Zug ausstieg, direkt in den nächsten lokalen Bus wollte und nicht gefunden habe, wo ich Bustickets kaufe. Es ist aktuell ein ziemliches Elend. ÖPNV-Unternehmen sind entweder gar nicht digital oder jedes hat seine eigene Ticket-App. Ich habe selbst 2021-2022 an der ÖPNV-App in Wien mitgewirkt und kenne die politischen Spiele, die dahinter stehen. Die Unternehmen wollen keine offenen Schnittstellen, sondern selber die Hand über dem Ticketverkauf haben, bauen gleichzeitig aber nicht die nötige Infrastruktur und Know-How auf um dies bequem anzubieten. Das Ergebnis ist, dass jede Stadt, jeder Landkreis, jedes Bundesland unterschiedliche und nicht-miteinander kompatible Apps verwenden. Als Reisender in dieses Trara reingeworfen zu werden ist Scheiße.

Nun war ich in Italien dieser Reisende und die Idee für Kebus kam auf.

Kebus ist eine kleine Web-App, die für deine aktuelle Position die verfügbaren Apps für den öffentlichen Personennahverkehr ausgibt. Derzeit sind etwa 50 Apps für über 60 Regionen enthalten. Das reicht von einzelnen Städten bis ganze Länder – hauptsächlich Europa. Derzeit ist es noch sehr kompakt, es spuckt nur die Apps für die aktuelle Position aus. Es gibt noch keine Suche und Ticketverkauf wird nie enthalten sein. Für mich war es mal wieder ein Pet-Projekt um einige Technologien zu testen. Der Aufwand liegt aber in der Pflege und das ist ohne lokales Wissen durchaus kompliziert. Wie es weitergeht hängt jetzt etwas davon ab, ob ich etwas Traction sehe.

Wenn in deiner Gegend noch keine App eingetragen ist, dann gerne Mail an mich – am Besten welche App für welche Region hinzugefügt werden soll.

Posted on

Everchords – The Story

Als ich 2014 mit Songwriting began hatte ich ein Problem. Ich lernte in dem ich Songs nachspielte und dafür die Texte und Akkorde nachspielte. An die zu kommen ist meistens nicht schwer, aber die sich auf eine kompakte und nicht-nervende Art anzeigen zu lassen war fast immer unmöglich. Seiten wie Ultimate Guitar, oder ähnliche haben zwar große Kataloge, aber die visuelle Aufarbeitung und UX ist oft mangelhaft. Liedtexte werden meist vertikal dargestellt, Akkorde als Zwischenzeilen – fast immer sprengen sie die Bildschirmhöhe, wodurch Scrollen nötig wird. Das versuchen manche durch Auto-Scroll zu kompensieren, aber geil ist es nicht. Will man die Texte mal ausdrucken ist das Resultat auch eher schwach.

In meinem Songwriting-Prozess, iteriere ich viel über meine Texte. Ich schreibe, ich spiele, ich ändere und das auch oft unterwegs. Tools wie Ukegeeks Song-O-Matic kann man auch selbst hosten und bieten gute Funktionen, aber leider fiel es mir schwer meine Songs zwischen der mobilen Notizen-App und dem Song Editor synchron zu halten. Es brauchte etwas, das beides verbindet.

2015 entstand die erste Version von Everchords, einem schlanken Frontend, dass Evernote als Song-Datenbank nutzt. Notizbücher in Evernotes, die meine Songs als Notizen enthalten werden in Everchords importiert und Everchords bietet dafür eine Oberfläche, diese Songs genau so zu formatieren wie ich es brauche. Ãœber die Zeit wuchs dies ein bisschen, aber nicht massiv. Das Tool sollte immer kompakt und den Focus auf die Song-Inhalte haben. Grifftabellen und Akkord-Diagramme kamen dennoch mit der Zeit dazu.

Die Anbindung von Evernote ist gleichzeitig die größte Stärke und die größte Hürde. Evernote ist zufällig mein Notizentool und bietet im Gegensatz zu vielen anderen eine gute API zum Synchronisieren von Inhalten. Allerdings ist Everchords ohne Evernote nicht benutzbar wodurch eine Nischanwendung noch weiter in seiner potenziellen Zielgruppe eingeschränkt wird. Dies ist letztlich auch der Hauptgrund warum ich es nie größer gezogen oder ein vollständiges Produkt draus gemacht habe. Dies ist ein Tool, für mich, für mein Songwriting und ein Pet-Projekt um bei Programmieren nicht ganz einzurosten.

Warum nun schreib ich davon: Everchords ist nun auf www.everchords.app dennoch online gegangen und ich habe mich entschieden es schließlich doch voll öffentlich for anyone zu machen. Enjoy.

Posted on

Spiele-Empfehlungen Pre-2020

Schon vor der Pest war ich ein Spiele-Fan und habe die Jahre mit diversen Indie- und Coop-Games verbracht. Da meine letzte Games-Liste gut angenommen wurde, ist hier Teil 2 mit älteren Spielen, die mich vor 2020 begeistert haben.

Wieder gilt, es ist ein kleiner Hang zu Indie- und Story-Games. Die absoluten Blockbuster wie Legend of Zelda – Breath Of The Wild, lass ich hier aus.

Continue reading “Spiele-Empfehlungen Pre-2020”
Posted on

From the reddits — PM Q&A #3 Transition into PM

A lonely bike on a beach in Myanmar.

Over the past year I’ve been spending some time in the subreddit r/ProductManagement and with my background of 10 years in Software Development and 3 years in Product Management, I’m happy to write where I can contribute. In this series, I’ll republish some of the responses I’ve written and maybe expand on them. This is part three, also checkout Part One on Team Culture and Two on Company Mergers.

Continue reading “From the reddits — PM Q&A #3 Transition into PM”