EJB3, Stateful Firewalls, and Connection Timeouts with Payara

Recently, I encountered an interesting and frustrating problem when troubleshooting an EJB3 connection issue between two Payara server instances. After a change in firewall vendor, we noticed that remote EJB3 calls would start hanging after a period of inactivity. Initially, it seemed like an intermittent issue — but the deeper I looked, the more it felt like it never should have worked before the move either - a lovely Schroedinbug.

In this post, I’ll walk through the issue, why it happened, and the solution that ultimately fixed it.

The Problem

Our system uses remote EJB3 calls between two Payara instances, communicating over a network with a stateful firewall in between. After the firewall vendor change, we started seeing EJB3 calls hang indefinitely if the connection had been idle for some time. Once the hang occurred, it would eventually trigger client-side timeouts, retries, and application instability.

Here’s a simple view of the setup:

Architecture Diagram

» Continue reading


Avoiding passlib Dependency in Ansible When Generating Traefik User Files

I use Ansible to manage my server configuration, including copying over version-controlled docker-compose files and application configuration. One such configuration is for Traefik, where I maintain a user file for HTTP basic authentication. These credentials are sourced from 1Password at deploy time and stored in a format compatible with htpasswd.

The Original Approach

Previously, I used the ansible.builtin.htpasswd module like so:


    - name: Create traefik user file
      ansible.builtin.htpasswd:
        path: "{{ traefik_conf_dir }}/usersfile"
        name: "{{ lookup('onepassword', 'Traefik', field='username') }}"
        password: "{{ lookup('onepassword', 'Traefik', field='password') }}"
        owner: "{{ ansible_user }}"
        group: "{{ ansible_user }}"
        mode: 0644

However, after upgrading to TrueNAS 25.04 “Fangtooth”, this failed with the following error:


TASK [Create traefik user file] ********************************************************************
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: ModuleNotFoundError: No module named 'passlib'
fatal: [truenas]: FAILED! => {"changed": false, "msg": "Failed to import the required Python library (passlib) on truenas's Python /usr/bin/python3.11. Please read the module documentation and install it in the appropriate location. If the required library is installed, but Ansible is using the wrong Python interpreter, please consult the documentation on ansible_python_interpreter"}

This module requires the passlib library, which wasn’t available on the system. I didn’t want to modify the system Python installation just to resolve this, especially on an appliance OS like TrueNAS.

» Continue reading


LLMs on a Homelab Without a GPU? Here's What I Found

I’ve recently been experimenting more with the power of Large Language Models (LLMs) and how they can supercharge my productivity. This especially has helped with the problem of a blank page wondering where to start writing – I can dump my unsorted thoughts into a chatbot and get a skeleton of a document or implementation in a few minutes. It’s by no means perfect what comes out, but it’s (usually) a very good start. I’ve been playing with both ChatGPT, and a series of local models running on my laptop with Ollama, Fabric, and OpenWebUI.

I wanted to be able to continue to use some of these self-hosted models while I’m out on the go, so I wondered - could I host these on my homelab server? My main homelab server runs an AMD Ryzen 5600G processor which includes an integrated GPU. Can this be used to improve the performance of a local model when I do not have a dedicated GPU available?

That doesn’t seem possible right now though, at least not in an efficient way. This post is a point-in-time snapshot of what I found when looking in to this.

» Continue reading


Installing Python Libraries from Source: pip, Poetry, and Flat Layout Pitfalls

There may be times when you want to install a Python library from source — maybe to test a specific commit, use a pre-release version, or debug a branch. Normally, this is relatively straightforward:

pip install git+https://github.com/aws-ia/taskcat@73de28457c66635a4517393c86484c18f44fef6f

But sometimes, things aren’t that easy… we are getting an error when trying to do this:

  error: Multiple top-level packages discovered in a flat-layout: ['e2e', 'assets', 'taskcat', 'installer', 'taskcat_plugin_testhook'].

  To avoid accidental inclusion of unwanted files or directories,
  setuptools will not proceed with this build.

  If you are trying to create a single distribution with multiple packages
  on purpose, you should not rely on automatic discovery.
  Instead, consider the following options:

  1. set up custom discovery (`find` directive with `include` or `exclude`)
  2. use a `src-layout`
  3. explicitly set `py_modules` or `packages` with a list of names

  To find more information, look for "package discovery" on setuptools docs.

» Continue reading


Modernising the Experiments Project - Containerisation and Code Coverage Improvements

Wow, it’s been over 4 years since the last post in this series. At that point I had hit all the key features that I thought I needed to learn for an upcoming project, and then moved on to other things. Now I’m coming up to some work where I may need this sample application for looking into aspects of OpenTelemetry - so it is time for some updates.

The microprofile-experiments project was really out of date. This PR brings the project up to the latest versions of most dependencies, but also includes some notable improvements.

The PR looks bigger than the changes really are however, mostly due to a combination of the javax to jakarta namespace change and the code formatter setup altering the expected import order.

» Continue reading