Jekyll2021-12-29T22:49:24+00:00https://blog.dwagenk.com/feed.xmlDaniel WagenknechtBlog about embedded systems and other technology that interests meThe Generalist’s Curse2021-12-29T00:00:00+00:002021-12-29T00:00:00+00:00https://blog.dwagenk.com/debugging/2021/12/generalists-curse<p>It’s time for another blog post. I’ve been working on a side project over the
past couple of weeks. It goes by the preliminary name <code class="language-plaintext highlighter-rouge">kodinix</code> and is yet
another incarnation of the kodi media center software on a small single board
computer (SBC). It is similar to <a href="https://libreelec.tv/">LibreELEC</a>, but adding
a crucial bit of functionality: the ability to open a browser as fallback for
accessing web content that is not covered by a kodi plugin. More details will
follow shortly (the usual promise :wink:).</p>
<p>In this context I had to debug some issues with the graphics stack that drove
me nuts for a few days. Working with embedded linux systems both professionally
and for fun I often touch a wide variety of software components. Each and every
one of them is a complex beast by itself. There’s people and teams out there
for whom the development and maintenance for any single of these projects is a
full-time job spanning years to decades, they are without doubt experts in
their domain. Stitching together those components into a full system,
selecting, packaging and confgiuring them to work well together is a task that
leaves me with a glimpse into a wide variety of components and technologies,
but for very few of these I have deeper knowledge or insight. I’d thus consider
myself a generalist. And that sometimes makes it hard to track down issues.</p>
<!--more-->
<p>The generalist’s curse is that the toes are far out in unknown waters when
tracking down an issue. Each of the many ponds around us could hold the key for
solving it. But getting acquainted to one pond, aquiring the needed
throughsight just to find out that the problem lays somewhere else takes a lot
of energy and time. And this can be frustrating as hell.</p>
<p>For this project I’m using a Pine64 H64 (model B) board. It’s a nicely specced
board and should be able to suffice for video streaming. But I couldn’t even
get it to load th kodi GUI. Well no, that’s not true. It did. At first. But
than it didn’t anymore. So I reverted to the previous NixOS configuration…</p>
<p>Aaaaaand</p>
<p>… it still didn’t work. Down the rabbit hole I went.</p>
<p>To let you parttake in this I’ll elaborate on the relevant bits of the tech
stack a little. We’ve got:</p>
<ul>
<li>the Pine64 H64 board
<ul>
<li>ARM SoC</li>
<li>ARM Mali T720 GPU</li>
<li>Display connected via HDMI</li>
</ul>
</li>
<li>mainline 64-bit ARM (aarch64) Linux kernel (v 5.10)</li>
<li>a 32-bit ARM (armv7l-hf) userland</li>
<li>Mesa3D graphics libs with panfrost GPU driver</li>
<li>cage wayland compositor</li>
<li>kodi media center software using wayland backend</li>
<li>kodi browser launcher plugin</li>
<li>firefox web browser</li>
</ul>
<p>“Why mix architectures?” you might ask. Well, I’m stubborn and don’t want to
run a 32bit kernel on hardware that can do better. Besides, I don’t really know
if building a 32bit Linux kernel for this SoC and board would even properly
work since some relevant device-tree bits live in arm64 subdirs of the kernel
source tree. The 32bit userland is needed because I want widevine to work, a
proprietary library needed to play DRM<sup id="fnref:DRM" role="doc-noteref"><a href="#fn:DRM" class="footnote" rel="footnote">1</a></sup> <del>protected</del> restricted media.
Google doesn’t really release this library for ARM architectures, but their
chromebooks ship a 32-bit ARM variant. There’s archane knowledge and utilities
in the kodi community to extract the library from chromebook recovery images
and use it inside kodi.</p>
<p>The problems manifested after small changes and a reboot. The graphics crashed
whenever the cage compositor was started. The TTY1 console was displayed fine
though. Reverting to the previous configuration didn’t bring the system back
into a working state. This got me off track, since NixOS manages the system
configuration in a declarative manner and thus should prevent exactly these
runtime inconsistencies. But I hadn’t done any deeper inspection of the
graphics or the general system before this error occured, so I was also lacking
the reference to compare against.</p>
<p>I first blamed mesa and the panfrost GPU driver for this. The panfrost driver
is still quite young. Mesa as a system library linking to hardware specific
backends is complex and not 100% pure even in NixOS. Enough clues to start
digging here. I spent some time recompiling mesa, ensuring it provided the
panfrost driver and left out unrelated drivers. In the meantime I read up on
mesa’s docs and the development state of the panfrost driver. By the time
everything was ready for testing I had almost forgotten what the next planned
step was. The long compile times of system components make it hard to stay
focused on the assumption that I’m currently evaluating. I find hand-written
notes quite helpful to keep some structure with this, whilest digital
notekeeping add even more context switches and thus make things even worse.</p>
<p>I figured out it wasn’t panfrosts fault. Starting the system with a software
rendering driver yielded the same result. Mesa’s documentation listed some
environment variables that increased verbosity. The wall of text hinted at a
problem with buffer allocation. After some research I tried increasing the
memory region reserved for the kernels ContiniousMemoryAllocator (CMA) by
adding a <a href="https://www.kernel.org/doc/html/v5.10/admin-guide/kernel-parameters.html?highlight=cma"><code class="language-plaintext highlighter-rouge">cma=</code>
argument</a>
to the kernel cmdline. It didn’t work and I moved on. In hindsight this was the
correct approach probably just carried out wrong.</p>
<p>I trimmed down my test setup a bit, experienced the same issue with test
programs like kmscube and started inspection of why I couldn’t get any core
dumps (probably due to systemd-coredumpd and the kernel being incompatible due
to the architecture mismatch). Many time consuming dead ends. At some
intermediary point I had it mostly working again by hard coding the
ExtendedDisplayIdentificationData (EDID) to use a lower display resolution. The
graphics didn’t crash immediately anymore, the crash was delayed to some random
point later in time. This and the fact that the crashlogs - where present -
still mentioned buffer allocation errors made me revisit the CMA a few days
later. Despite having tried it before. This time it worked.</p>
<p>I cursed. And I was happy to finally have a solution to the problem. I’ve
learned some more about the linux graphics stack and especially mesa in the
meantime, but I still feel like the next similar poroblem will probably be way
above my head again. This is the curse of the generalist…</p>
<hr />
<p>Feel free to contact me via <a href="mailto:dwagenk@mailbox.org">mail</a> or
<a href="https://chaos.social/@dwagenk">mastodon</a> if you’ve got any notes on this post
or start a public discussion via this blogs <a href="https://github.com/dwagenk/blog.dwagenk.com/issues">github issue
tracker</a>. You can also
subscribe to the <a href="/feed.xml">RSS feed</a> to stay tuned!</p>
<div class="footnotes" role="doc-endnotes">
<ol>
<li id="fn:DRM" role="doc-endnote">
<p>Speaking of DigitalRightsManagement aka DigitalRestrictionsManagement here. The linux kernel graphics stack also includes a component named DRM that can lead to a little confusion. <a href="#fnref:DRM" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
</ol>
</div>It’s time for another blog post. I’ve been working on a side project over the past couple of weeks. It goes by the preliminary name kodinix and is yet another incarnation of the kodi media center software on a small single board computer (SBC). It is similar to LibreELEC, but adding a crucial bit of functionality: the ability to open a browser as fallback for accessing web content that is not covered by a kodi plugin. More details will follow shortly (the usual promise :wink:). In this context I had to debug some issues with the graphics stack that drove me nuts for a few days. Working with embedded linux systems both professionally and for fun I often touch a wide variety of software components. Each and every one of them is a complex beast by itself. There’s people and teams out there for whom the development and maintenance for any single of these projects is a full-time job spanning years to decades, they are without doubt experts in their domain. Stitching together those components into a full system, selecting, packaging and confgiuring them to work well together is a task that leaves me with a glimpse into a wide variety of components and technologies, but for very few of these I have deeper knowledge or insight. I’d thus consider myself a generalist. And that sometimes makes it hard to track down issues.Canceling the #15WeeksToBlog Challenge2020-06-08T00:00:00+00:002020-06-08T00:00:00+00:00https://blog.dwagenk.com/meta/2020/06/canceling-15WeeksToBlog<p>Hey there! I’ve decided to cancel #15WeeksToBlog challenge I set for myself to
stay motivated to blog more often. I started it whilst the
#100DaysToBlog / #100DaysToOffload challenges trended on mastodon, knowing very
well that I don’t want additional incentives for spending time in front of a
screen <strong>daily</strong> and writing something supposedly “offloading”. The
spirit pushed me towards reviving my own blog though and I tried the weekly
blog post to find a fitting level between regular posts and the time and
resources to give them enough depth so I don’t see them as irrelevant and “just
another guys random thoughts on the internet”. I’ve steadily but slowly been
working on blog posts since then, but time is even more sparse than I expected
(between the covid19-pandemic related inconveniences, some contract work,
searching for a more permanent job, arriving and furnishing the new apartment
and taking care of three kids bustling around the home), so weekly and even
bi-weekly posts is nothing I can commit to right now.</p>
<p>There’s more posts about Nix and NixOS in the pipeline and I also want to
document my recent encounters building a small multi-apartment network based on
OpenWRT enabled hardware. They’ll be published when they’re done. Might be next
week, but could just as well be next month. So give this blog an occasional
visit or subscribe to the <a href="/feed.xml">RSS feed</a> if you’re interested in those
topics. Thanks!</p>
<hr />
<p>Feel free to contact me via <a href="mailto:dwagenk@mailbox.org">mail</a> or
<a href="https://chaos.social/@dwagenk">mastodon</a> if you’ve got any notes on this post
or start a public discussion via this blogs <a href="https://github.com/dwagenk/blog.dwagenk.com/issues">github issue
tracker</a>. You can also
subscribe to the <a href="/feed.xml">RSS feed</a> to stay tuned!</p>Hey there! I’ve decided to cancel #15WeeksToBlog challenge I set for myself to stay motivated to blog more often. I started it whilst the #100DaysToBlog / #100DaysToOffload challenges trended on mastodon, knowing very well that I don’t want additional incentives for spending time in front of a screen daily and writing something supposedly “offloading”. The spirit pushed me towards reviving my own blog though and I tried the weekly blog post to find a fitting level between regular posts and the time and resources to give them enough depth so I don’t see them as irrelevant and “just another guys random thoughts on the internet”. I’ve steadily but slowly been working on blog posts since then, but time is even more sparse than I expected (between the covid19-pandemic related inconveniences, some contract work, searching for a more permanent job, arriving and furnishing the new apartment and taking care of three kids bustling around the home), so weekly and even bi-weekly posts is nothing I can commit to right now.Diving into NixOS: Scanning2020-05-20T00:00:00+00:002020-05-20T00:00:00+00:00https://blog.dwagenk.com/nix/2020/05/nix-scanning<p>I announced posting more about my journey getting familiar with <em>Nix/NixOS</em> and
getting my printer/scanner combo device to work well with it. Setting myself a
challenge to keep blogging weekly
(<a href="https://chaos.social/web/timelines/tag/15WeeksToBlog">#15WeeksToBlog</a>) and
seeing a lot of other peoples blog posts appearing has motivated me enough to
actually get this post written down albeit a lot later than planned.</p>
<!--more-->
<p>Like mentioned in the <a href="/nix/2020/04/nix-printing">post about printing</a> I’ve got
a <em>Canon PIXMA TR4551</em> printer/scanner combo device with USB and Wi-Fi
connectivity. I’ve used it for scanning with linux before, using <em>Canons</em>
proprietary scanning application <em>scangearmp2</em>. Using it is no fun, it doesn’t
remember any preferences like the preferred paper format. The worst part: the
save dialog window is unusable with the keyboard. It doesn’t preselect the file
name input field and the save button is absolutely inaccessible by keyboard.</p>
<blockquote>
<p>In most applications there’s underlined letters in every button, menu item
etc. when pressing the ALT key, that allow selecting them via the keyboard.
Alternatively they accept ENTER for the default action (OK/SAVE/CONFIRM) and
ESC to abort. This application does neither.</p>
</blockquote>
<p>The scanner has an automatic document feeder (ADF) that is supported by the
application, so scanning a multi-page document is OK, as long as it’s
single-sided and fits into the feeder.</p>
<p>Well, I guess the impression, why I’d rather use some “better” software got
across, so let’s boil this down into a list of requirements for a driver or
scanner-application:</p>
<ul>
<li>works via network / Wi-Fi</li>
<li>allows selection of default values (paper format, source, color-mode) or
remembers the last used options</li>
<li>doesn’t require manual input for the file name. A quick scan with a default
filename should not involve any other button presses the OK, OK, NEXT, SAVE,
OK, … (and not that many)</li>
<li>is usable without reaching for the mouse</li>
<li>supports the ADF</li>
<li>(optional) Scanning the next page of a multi-page document can be triggered
by pressing a button on the scanner</li>
<li>(optional) Text recognition (OCR)</li>
</ul>
<h1 id="sane">SANE</h1>
<p>The usual setup for scanning on linux involves the <em>SANE Project</em>, with the
acronym standing for <em>ScannerAccessNowEasy</em>. <em>SANE</em> has a huge selection of
backends — the device-specific drivers — and provides a common API for their
usage, so that the actual applications (the <em>SANE</em> frontends) are device
independent. Regarding the frontends I’ve used
<a href="http://gscan2pdf.sourceforge.net/">gscan2pdf</a> for documents and
<a href="https://gitlab.gnome.org/GNOME/simple-scan">SimpleScan</a> for quick scans of
single pages or photos in the past and the combination of those tick all the
frontend-related boxes of my requirements list from above. So all that’s
missing is a network-enabled <em>SANE</em> backend for my printer which also supports
the ADF and optionally can trigger the next scan via an on-device button.</p>
<h2 id="backends">Backends</h2>
<p>Online-research led to quite a few options with a varying amount of information
available:</p>
<table>
<thead>
<tr>
<th>Backend</th>
<th>included with <em>SANE</em> <br /> (version)</th>
<th>packaged for <em>Nix</em> <br /> (channel)</th>
<th>Wi-Fi</th>
<th>USB</th>
<th>ADF</th>
<th>on-device button</th>
</tr>
</thead>
<tbody>
<tr>
<td>eSCL</td>
<td>:heavy_check_mark: (1.0.29)</td>
<td>:x:</td>
<td>:heavy_check_mark:</td>
<td>:x:</td>
<td>:heavy_check_mark:</td>
<td>:x:</td>
</tr>
<tr>
<td>airscan</td>
<td>:x:</td>
<td>:heavy_check_mark: (unstable)</td>
<td>:heavy_check_mark:</td>
<td>:x:</td>
<td>:heavy_check_mark:</td>
<td>:x:</td>
</tr>
<tr>
<td>pixma</td>
<td>:heavy_check_mark:(1.0.28)</td>
<td>:heavy_check_mark: (20.03)</td>
<td>:grey_question:</td>
<td>:heavy_check_mark:</td>
<td>:heavy_check_mark:</td>
<td>:heavy_check_mark:</td>
</tr>
<tr>
<td>canon-pixma</td>
<td>:x:</td>
<td>:x:</td>
<td>:heavy_check_mark:</td>
<td>:heavy_check_mark:</td>
<td>:grey_question:</td>
<td>:grey_question:</td>
</tr>
</tbody>
</table>
<h3 id="pixma">pixma</h3>
<p>Support for my scanner via the <em>pixma</em> backend might work. The list of features
that are supported in general, but might or might not be
supported for my device, is long, including niceties like button controlled
scanning, adjusting image settings (gamma, brightness) or setting a wait time
for when the automated document feeder runs empty, to allow reloading it and
continue scanning the same document. The documentation mentioned networked
scanning support via the proprietary <em>Canon BJNP</em> protocol, but I didn’t find
any information, whether this was supported by my device.</p>
<p>In recent versions of the <a href="http://www.sane-project.org/sane-mfgs.html#Z-CANON"><em>SANE</em> list of supported
hardware</a> the device is
listed as <code class="language-plaintext highlighter-rouge">untested</code> meaning its information has been added to the backend, to
enable testing of the status of device-support without further modification to
the source code. My switch to the <code class="language-plaintext highlighter-rouge">nixos-20.03</code> channel containing the required
<em>SANE</em> package was due anyhow, so I switched channels, waited for the system to
update and installed <em>SANE</em>.</p>
<p>The scanner wasn’t found via the network even with temporarily disabled
firewall, so the <em>pixma</em> backend won’t work for me. I did a quick check of all
the other functions with the scanner connected via USB. Scanning from the
flatbed worked, but I got an error when trying to use the ADF. This is now
reported back to the <em>SANE</em> project.</p>
<blockquote>
<p>If you encounter bugs or other annoyances in openly
developed software, please do file a bug, provide fixes or suggest improvements
to the documentation (some projects pay a lot of attention to keeping their
barrier of entry low and their communication tone very welcoming, to make it
easy for people with and without coding skills to contribute).</p>
</blockquote>
<h3 id="escl-and-airscan">eSCL and airscan</h3>
<p>The eSCL and airscan backends both use the HTTP-based eSCL<sup id="fnref:escl" role="doc-noteref"><a href="#fn:escl" class="footnote" rel="footnote">1</a></sup> protocol and
don’t need device-specific drivers. The protocol seems to be well known through
<em>Apples</em> use of it and is supported by a quite broad palette of network
connected scanners, but it seems to not be publicly documented. Work on
reverse-engineering and reimplementing (parts of) it in free software drivers
is has just started recently, with most documentation notes and work on the SANE backends
dating to 2019.</p>
<p>Details on how to get those backends working in <em>Nix</em> will be part of a separate
blog post, but I sure was surprised about the amount of work that can be
associated with bumping up a packages version number.</p>
<p>Using the airscan backend scanner discovery, scanning from flatbed and ADF all
work<sup id="fnref:sane-airscan-error" role="doc-noteref"><a href="#fn:sane-airscan-error" class="footnote" rel="footnote">2</a></sup>, so the remaining parts of my requirements list are
covered and I guess I could stop at this point. But I’m trying to get an
overview of the available options here and experiment with <em>Nix</em> packaging, so
I can just as well go on and evaluate the remaining backends.</p>
<p>The eSCL backend has been a moving target while preparing this blog post. The
version contained in release 1.0.29 of the <em>SANE</em> backends was unpolished,
selecting the correct paper-size was buggy and it didn’t support the ADF yet.
This changed as development went on and the most recent commit fixing some
remaining ADF errors happened just a few days ago. Scanning from flatbed and ADF
work when using the <em>SimpleScan</em> GUI application, but I get messed up image
files when scanning via the commandline with the <em>scanimage</em> tool. I did open
another bug report for this.</p>
<h3 id="canon-pixma">canon-pixma</h3>
<p>This backend is written as a wrapper around the proprietary driver libraries,
that are part of <em>scangearmp2</em>. The main contributor to this wrapper is also
heavily involved in the <em>SANE</em> project, but the backend is not integrated into
the <em>SANE</em> backends repository, I guess that’s due to licencing issues. As it
isn’t packaged for <em>NixOS</em> yet, I had a little work to do here and write my
first package definition in <em>Nix</em>. I’m still working on polishing it for a pull
request_ to the <em>nixpkgs</em> repository, but for local testing it works already. I
also packaged the <em>scangearmp2</em> application, to have a first
check, if the libraries and the printer work together. <em>scangearmp2</em> didn’t
find the scanner via Wi-Fi at first, but disabling the firewall made it work and
there’s no difference between the scanner connected via USB and via Wi-Fi then.
For further testing I used the USB connection and reenabled the firewall, but
finding the correct ports to open in the firewall is still on my ToDo list. All
functions provided by <em>scangearmp2</em> seem to work, including scanning from the
ADF.</p>
<p><em>SANE</em> with the <em>canon-pixma</em> backend finds the scanner via USB and — with
disabled firewall — via Wi-Fi as well. Scanning works fine from the flatbed, but
the ADF doesn’t do anything. I haven’t yet found the cause for that and given that
most of the functionality is hidden in proprietary, binary-only libraries, the
debuggability is not so good. I’ll see, if I find the time and motivation to
dig deeper into that phenomenen, but since my needs are covered by other
backends and there’s no additional features to be expected to be usable via
this backend, I probably won’t waste a lot more time on this.</p>
<h2 id="verdict">Verdict</h2>
<p>To sum up, here’s the list of supported features again, based on what I
actually encountered:</p>
<table>
<thead>
<tr>
<th>Backend</th>
<th>included with <em>SANE</em> <br /> (version)</th>
<th>packaged for <em>Nix</em> <br /> (channel)</th>
<th>Wi-Fi</th>
<th>USB</th>
<th>ADF</th>
<th>on-device button</th>
</tr>
</thead>
<tbody>
<tr>
<td>eSCL</td>
<td>:beetle: (1.0.29) <br /> :heavy_check_mark: (master)</td>
<td>:x:</td>
<td>:heavy_check_mark:</td>
<td>:x:</td>
<td>:heavy_check_mark:</td>
<td>:x:</td>
</tr>
<tr>
<td>airscan</td>
<td>:x:</td>
<td>:heavy_check_mark: (unstable)</td>
<td>:heavy_check_mark:</td>
<td>:x:</td>
<td>:heavy_check_mark:</td>
<td>:x:</td>
</tr>
<tr>
<td>pixma</td>
<td>:heavy_check_mark:(1.0.28)</td>
<td>:heavy_check_mark: (20.03)</td>
<td>:x: or :beetle:</td>
<td>:heavy_check_mark:</td>
<td>:beetle:</td>
<td>:beetle:</td>
</tr>
<tr>
<td>canon-pixma</td>
<td>:x:</td>
<td>:x:</td>
<td>:heavy_check_mark:</td>
<td>:heavy_check_mark:</td>
<td>:beetle:</td>
<td>:x:</td>
</tr>
</tbody>
</table>
<p>I’ll be using the airscan backend for now, since it is packed for Nix and can
easily be integrated into my system without me having to keep local package
definitions around.</p>
<hr />
<p>This post is part of the
<a href="https://chaos.social/web/timelines/tag/15WeeksToBlog">#15WeeksToBlog</a>
challenge.<br /> Week 2/15. Technically I failed the challenge already, skipping
more than a. I’ve been steadily working on the post though, ensuring my notes
from the experiments a little while ago where still correct. The backends
getting updated in the meantime led to me reevaluating some aspects. I’ll
interpret this experience as a push toward smaller, more focused posts and
publishing them faster.</p>
<p>Feel free to contact me via <a href="mailto:dwagenk@mailbox.org">mail</a> or
<a href="https://chaos.social/@dwagenk">mastodon</a> if you’ve got any notes on this post
or start a public discussion via this blogs <a href="https://github.com/dwagenk/blog.dwagenk.com/issues">github issue
tracker</a>. You can also
subscribe to the <a href="/feed.xml">RSS feed</a> to stay tuned!</p>
<div class="footnotes" role="doc-endnotes">
<ol>
<li id="fn:escl" role="doc-endnote">
<p>eSCL actually contains both, Apple AirScan and Apple AirPrint <a href="#fnref:escl" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:sane-airscan-error" role="doc-endnote">
<p>It reports an error when the ADF runs empty. The scanned documents are successfully retrieved though. <a href="#fnref:sane-airscan-error" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
</ol>
</div>I announced posting more about my journey getting familiar with Nix/NixOS and getting my printer/scanner combo device to work well with it. Setting myself a challenge to keep blogging weekly (#15WeeksToBlog) and seeing a lot of other peoples blog posts appearing has motivated me enough to actually get this post written down albeit a lot later than planned.Linking this blog with my mastodon profile2020-05-01T00:00:00+00:002020-05-01T00:00:00+00:00https://blog.dwagenk.com/fediverse/2020/05/mastodon-rel-me-link<p>Alongside the revival of this blog through yesterdays post I also added a link
to my mastodon profile in its footer. My mastodon profile also had a link to
this blog in it for a long time, though it missed the little “verified”
checkmark next to it. In case you’ve not seen the mark I’m referring to or are
not present on mastodon / the fediverse yet (you should join, e.g.
<a href="https://fosstodon.org">here</a> or <a href="https://social.tchncs.de">here</a>), you can see
it in this screenshot:</p>
<p><img src="/assets/2020-04-30-linking-mastodon-verified-screenshot.png" alt="Screenshots of verified blog address in tusky android
app" /></p>
<p>Getting it in there isn’t complicated and there’s several <a href="https://docs.joinmastodon.org/user/profile/">online
resources</a> on achieving that.
Basically it’s just adding a URL in one of the mastodon profile metadata fields
and then linking back to your mastodon profile on that website with a
<code class="language-plaintext highlighter-rouge">rel="me"</code> attribute.</p>
<div class="language-html highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt"><a</span> <span class="na">href=</span><span class="s">"https://chaos.social/@dwagenk"</span> <span class="na">rel=</span><span class="s">"me"</span><span class="nt">></span>My Mastodon Profile<span class="nt"></a></span>
</code></pre></div></div>
<p>One thing I didn’t see mentioned anywhere: The link verification doesn’t appear
immediately, but needs some time. Apparently the mastodon instance server scans
those URLs periodically instead of every time the profile is viewed (sounds
like a reasonable design decision favoring lower load times). This irritated
me, because it needed more than 9 hours for the checkmark to appear on my
profile. The timespan got me thinking and double checking, whether I made any
mistakes linking back, but in the end waiting was just the best solution for
this problem…</p>
<!--more-->
<hr />
<p>This post is part of the
<a href="https://chaos.social/web/timelines/tag/15WeeksToBlog">#15WeeksToBlog</a>
challenge.<br /> Week 1/15, 2nd post this week.</p>
<p>Feel free to contact me via <a href="mailto:dwagenk@mailbox.org">mail</a> or
<a href="https://chaos.social/@dwagenk">mastodon</a> if you’ve got any notes on this post
or start a public discussion via this blogs <a href="https://github.com/dwagenk/blog.dwagenk.com/issues">github issue
tracker</a>. You can also
subscribe to the <a href="/feed.xml">RSS feed</a> to stay tuned!</p>Alongside the revival of this blog through yesterdays post I also added a link to my mastodon profile in its footer. My mastodon profile also had a link to this blog in it for a long time, though it missed the little “verified” checkmark next to it. In case you’ve not seen the mark I’m referring to or are not present on mastodon / the fediverse yet (you should join, e.g. here or here), you can see it in this screenshot: Getting it in there isn’t complicated and there’s several online resources on achieving that. Basically it’s just adding a URL in one of the mastodon profile metadata fields and then linking back to your mastodon profile on that website with a rel="me" attribute. <a href="https://chaos.social/@dwagenk" rel="me">My Mastodon Profile</a> One thing I didn’t see mentioned anywhere: The link verification doesn’t appear immediately, but needs some time. Apparently the mastodon instance server scans those URLs periodically instead of every time the profile is viewed (sounds like a reasonable design decision favoring lower load times). This irritated me, because it needed more than 9 hours for the checkmark to appear on my profile. The timespan got me thinking and double checking, whether I made any mistakes linking back, but in the end waiting was just the best solution for this problem…Diving into NixOS: Printing2020-04-30T00:00:00+00:002020-04-30T00:00:00+00:00https://blog.dwagenk.com/nix/2020/04/nix-printing<p>I like to dive into problems to learn new skills. Especially when it comes to
technology and crafts. This led to a self-made cargo-bike, basic welding
skills, but also the basis for my profession: knowledge of embedded systems,
programming in C and configuring various software systems.</p>
<p>In this and probably two future blog posts I want to document my last deep-dive
into a new-to-me technology:</p>
<p><em>Nix</em>/<em>NixOS</em> or more specifically getting my printer/scanner combo device to work
well with <em>NixOS</em>.</p>
<!--more-->
<p><em>Nix</em> is a functional approach to packaging software, and approaching system
configuration. The OS is configured declaratively by writing config files,
instead of changing the system in an iterative way. I’ve used both <em>Debian</em> and
<em>Arch</em> <em>Linux</em> based distributions before and experienced problems regarding
software dependencies and malfunctioning packages now and then with them.
Probably <em>NixOS</em> will have some challenges for me as well, but I’m more willing
to tackle them. Whatever I do to mitigate those is documented and version
controlled in my system configuration and won’t come back to haunt me (<em>ooh, I
remember I spent hours debugging and solving this problem, but what the heck
did I do?</em>). I’ve had <em>NixOS</em> installed on my personal laptop for 3 months,
with barely enough configuration for it to serve my everyday needs, but without
wrapping my head around it yet. So it’s about time to develop better skills
around the system I’m using.</p>
<p>I’ll use this post to document the chunk that worked quite flawlessly:
getting the printer to work. I’ll have a similar post for the scanner as well,
which was more challenging to get running well.</p>
<p>I’ve bought a scanner/printer combo device (<em>Canon PIXMA TR4551</em>) with USB and
Wi-Fi connectivity about one and a half years ago. It worked OK with my
previous <em>Arch</em> <em>Linux</em> setup but needed proprietary software and drivers provided
by the vendor.</p>
<h2 id="ipp-everywhere">IPP Everywhere</h2>
<p>Before installing those drivers again on <em>NixOS</em>, I tried out using the printer
with available free software drivers. The printer supports <em>IPP</em>
(InternetPrintingProtocol) for access via the network. <em>CUPS</em>
(CommonUnixPrintingService, the standard way to access printers on <em>Linux</em>) has a
built-in <code class="language-plaintext highlighter-rouge">Generic IPP Everywhere Printer</code> driver. To get
this working on <em>NixOS</em> I added the line</p>
<div class="language-nix highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">services</span><span class="o">.</span><span class="nv">printing</span><span class="o">.</span><span class="nv">enable</span> <span class="o">=</span> <span class="kc">true</span><span class="p">;</span>
</code></pre></div></div>
<p>to my <code class="language-plaintext highlighter-rouge">configuration.nix</code> file.</p>
<p>Turns out this driver works OK with my printer, but there are some pitfalls:</p>
<ul>
<li>The driver allows selecting the print quality in DPI from a drop-down with
many many options. Selecting a value that’s not well-supported by my printer
causes no error message, but weirdly scaled pages</li>
<li>A similar issue arises with the color-profile selection</li>
<li>The duplex printing function is inversed</li>
</ul>
<p>Those pitfalls are more or less cosmetic, but would probably be a constant
source of failed printouts in the future. I’m human and tend to forget
which options I need to choose. I don’t want to be able to select unsupported
modes.</p>
<p>It’s nice to have the printer supported kinda-out-of-the-box. I decided to try
using <em>Canon</em>’s proprietary drivers as I remember them to work a little better.</p>
<h2 id="proprietary-driver">Proprietary Driver</h2>
<p>The drivers provided by <em>Canon</em> are already available as a package in <em>NixOS</em>!
The right keyword to use in the <em>Nix</em> package search is <em>cnijfilter2</em> which is
also part of the filename of the driver archive offered for download at
<em>Canon</em>’s website.</p>
<p>I went on extending my <code class="language-plaintext highlighter-rouge">configuration.nix</code> file like suggested in the <a href="https://nixos.wiki/wiki/Printing"><em>NixOS</em>
wiki</a></p>
<div class="language-nix highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">services</span><span class="o">.</span><span class="nv">printing</span><span class="o">.</span><span class="nv">enable</span> <span class="o">=</span> <span class="kc">true</span><span class="p">;</span>
<span class="nv">services</span><span class="o">.</span><span class="nv">printing</span><span class="o">.</span><span class="nv">drivers</span> <span class="o">=</span> <span class="p">[</span> <span class="nv">pkgs</span><span class="o">.</span><span class="nv">cnijfilter2</span> <span class="p">];</span>
</code></pre></div></div>
<p>This led to an error while rebuilding the <em>NixOS</em> system:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ sudo nixos-rebuild switch
building Nix...
building the system configuration...
error: Package ‘cnijfilter2-5.30’ in /nix/store/ywjr6djpwc1cp4hp52yqwwgyb9vphkql-nixos-19.09.2426.9237a09d8ed/nixos/pkgs/misc/cups/drivers/cnijfilter2/default.nix:117 has an unfree license (‘unfree’), refusing to evaluate.
a) For `nixos-rebuild` you can set
{ nixpkgs.config.allowUnfree = true; }
in configuration.nix to override this.
b) For `nix-env`, `nix-build`, `nix-shell` or any other _Nix_ command you can add
{ allowUnfree = true; }
to ~/.config/nixpkgs/config.nix.
(use '--show-trace' to show detailed location information)
</code></pre></div></div>
<p>following the hint in the error message and adding</p>
<div class="language-nix highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">nixpkgs</span><span class="o">.</span><span class="nv">config</span><span class="o">.</span><span class="nv">allowUnfree</span> <span class="o">=</span> <span class="kc">true</span><span class="p">;</span>
</code></pre></div></div>
<p>to <code class="language-plaintext highlighter-rouge">configuration.nix</code> solved the problem. The printer works with the new
driver and I’m glad it offers fewer options and therefore fewer possibilities
for me to screw up, but […]</p>
<h2 id="a-note-on-free-software">A Note on Free Software</h2>
<p>[…] I don’t like the <code class="language-plaintext highlighter-rouge">allowUnfree = true</code> setting. I’m a big fan of free
software and use it whenever there’s a viable solution available. Being
pragmatic I use proprietary software on my devices as well. The decision should
remain on a per-package base though, not to give carte blanche for all unfree
packages. I found the relevant
<a href="https://nixos.org/nixpkgs/manual/#sec-allow-unfree">section</a> in the nixpkgs
manual, but the solution presented there didn’t work. Turns out there’s also a
<a href="https://github.com/NixOS/nixpkgs/issues/67830#issuecomment-542933255">github
issue</a>
mentioning the documentation being incorrect and offering a working alternative
for allowing individual unfree packages.</p>
<div class="language-nix highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">nixpkgs</span><span class="o">.</span><span class="nv">config</span><span class="o">.</span><span class="nv">allowUnfreePredicate</span> <span class="o">=</span> <span class="p">(</span><span class="nv">pkg</span><span class="p">:</span> <span class="kr">builtins</span><span class="o">.</span><span class="nv">elem</span>
<span class="nv">pkg</span><span class="o">.</span><span class="nv">pname</span> <span class="p">[</span>
<span class="s2">"cnijfilter2"</span>
<span class="p">]</span>
<span class="p">);</span>
</code></pre></div></div>
<h2 id="network-printing">Network Printing</h2>
<p>The printer offers USB and Wi-Fi as means of communication. Using the
<em>cnijfilter2</em> driver <em>CUPS</em> finds the printer via it’s USB connection, but not
via Wi-Fi. Printing via Wi-Fi works after configuring the printer manually
(<code class="language-plaintext highlighter-rouge">ipp://123.ip.of.printer</code>). It would be nice to have <em>CUPS</em> auto-detect
network printers. Without going into detail (I really don’t know a lot about
auto-discovery of networked devices and just followed the clues from the
<em>NixOS</em> wiki) here’s what I had to add to my <code class="language-plaintext highlighter-rouge">configuration.nix</code> for <em>CUPS</em> to
automatically find the printer via Wi-Fi:</p>
<div class="language-nix highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">services</span><span class="o">.</span><span class="nv">avahi</span><span class="o">.</span><span class="nv">enable</span> <span class="o">=</span> <span class="kc">true</span><span class="p">;</span>
<span class="nv">services</span><span class="o">.</span><span class="nv">avahi</span><span class="o">.</span><span class="nv">nssmdns</span> <span class="o">=</span> <span class="kc">true</span><span class="p">;</span>
</code></pre></div></div>
<blockquote>
<p>Note: the printer needs to be in the same IP-subnet as the computer for
auto-discovery to work.</p>
</blockquote>
<h2 id="declarative-printer-configuration">Declarative Printer Configuration</h2>
<p>I’ve skipped over the part of actually configuring the printer, since I’ve not
done it <em>the Nix way</em> yet. Configuring printers declaratively in <em>NixOS</em> seems
to be possible and I’ll take a look into it later.</p>
<h2 id="config-file">Config File</h2>
<p>To summarize, this is the section in my <code class="language-plaintext highlighter-rouge">configuration.nix</code> related to printing</p>
<div class="language-nix highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span>
<span class="c">#...</span>
<span class="nv">services</span><span class="o">.</span><span class="nv">printing</span><span class="o">.</span><span class="nv">enable</span> <span class="o">=</span> <span class="kc">true</span><span class="p">;</span>
<span class="nv">services</span><span class="o">.</span><span class="nv">printing</span><span class="o">.</span><span class="nv">drivers</span> <span class="o">=</span> <span class="p">[</span> <span class="nv">pkgs</span><span class="o">.</span><span class="nv">cnijfilter2</span> <span class="p">];</span>
<span class="nv">nixpkgs</span><span class="o">.</span><span class="nv">config</span><span class="o">.</span><span class="nv">allowUnfreePredicate</span> <span class="o">=</span> <span class="p">(</span><span class="nv">pkg</span><span class="p">:</span> <span class="kr">builtins</span><span class="o">.</span><span class="nv">elem</span>
<span class="nv">pkg</span><span class="o">.</span><span class="nv">pname</span> <span class="p">[</span>
<span class="s2">"cnijfilter2"</span>
<span class="p">]</span>
<span class="p">);</span>
<span class="nv">services</span><span class="o">.</span><span class="nv">avahi</span><span class="o">.</span><span class="nv">enable</span> <span class="o">=</span> <span class="kc">true</span><span class="p">;</span>
<span class="nv">services</span><span class="o">.</span><span class="nv">avahi</span><span class="o">.</span><span class="nv">nssmdns</span> <span class="o">=</span> <span class="kc">true</span><span class="p">;</span>
<span class="c">#...</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Feel free to contact me via <a href="mailto:dwagenk@mailbox.org">mail</a> or
<a href="https://chaos.social/@dwagenk">mastodon</a> if you’ve got any notes on this post
or start a public discussion via this blogs <a href="https://github.com/dwagenk/blog.dwagenk.com/issues">github issue
tracker</a>. Like I mentioned
above, I’ve got more posts planned on <em>Nix</em> and my printer/scanner, so if this
interests you subscribe to the <a href="/feed.xml">RSS feed</a> to stay tuned!</p>
<p><a href="http://creativecommons.org/licenses/by-sa/4.0/">CC-BY-SA-4.0</a></p>I like to dive into problems to learn new skills. Especially when it comes to technology and crafts. This led to a self-made cargo-bike, basic welding skills, but also the basis for my profession: knowledge of embedded systems, programming in C and configuring various software systems. In this and probably two future blog posts I want to document my last deep-dive into a new-to-me technology: Nix/NixOS or more specifically getting my printer/scanner combo device to work well with NixOS.Setting up Nextcloud with Collabora Online Office using Docker2019-03-31T00:00:00+00:002019-03-31T00:00:00+00:00https://blog.dwagenk.com/2019/03/collabora-for-nextcloud<h2 id="prelude">Prelude</h2>
<p>I started writing this blog post in a first attempt a couple days ago and it got way too verbose and lengthy, going into detail about the specifics of my setup and the reasoning behind it. So here’s a short version.</p>
<h2 id="goal">Goal</h2>
<p>Adding Collabora to an existing Nextcloud instance to enable online collaborative document editing right inside the Nextcloud webinterface.</p>
<h2 id="step-by-step">Step by Step</h2>
<h3 id="prerequisites">Prerequisites</h3>
<p>I’m assuming you’ve got Nextcloud installed somewhere already (publically reachable, with it’s own domain or subdomain) and also installed the Collabora app (it’s listed as one of the <em>official</em> apps in the <em>Office & Text</em> section of Nextcloud’s list of apps). You also need a server for running Collabora, that has docker and docker-compose installed and a domain or subdomain pointing to it. If both, Nextcloud and Collabora, will be running on the same server, take a closer look at the <code class="language-plaintext highlighter-rouge">jwilder/nginx-proxy</code>, that I’ll be using anyhow. It should be no problem having both running behind the proxy on the same server, accessible through different subdomains.</p>
<!--more-->
<h3 id="the-docker-composeyml-file">The <code class="language-plaintext highlighter-rouge">docker-compose.yml</code> file</h3>
<p>is stitched together from <a href="https://help.nextcloud.com/t/collabora-configuration-with-docker-compose/3970">this</a> thread:</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">version</span><span class="pi">:</span> <span class="s1">'</span><span class="s">2'</span>
<span class="na">services</span><span class="pi">:</span>
<span class="na">proxy</span><span class="pi">:</span>
<span class="na">image</span><span class="pi">:</span> <span class="s">jwilder/nginx-proxy</span>
<span class="na">container_name</span><span class="pi">:</span> <span class="s">proxy</span>
<span class="na">ports</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">80:80</span>
<span class="pi">-</span> <span class="s">443:443</span>
<span class="na">volumes</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">./proxy/conf.d:/etc/nginx/conf.d</span>
<span class="pi">-</span> <span class="s">./proxy/vhost.d:/etc/nginx/vhost.d</span>
<span class="pi">-</span> <span class="s">./proxy/html:/usr/share/nginx/html</span>
<span class="pi">-</span> <span class="s">./proxy/certs:/etc/nginx/certs:ro</span>
<span class="pi">-</span> <span class="s">/var/run/docker.sock:/tmp/docker.sock:ro</span>
<span class="na">networks</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">proxy-net</span>
<span class="na">letsencrypt-companion</span><span class="pi">:</span>
<span class="na">image</span><span class="pi">:</span> <span class="s">jrcs/letsencrypt-nginx-proxy-companion</span>
<span class="na">container_name</span><span class="pi">:</span> <span class="s">letsencrypt-companion</span>
<span class="na">volumes_from</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">proxy</span>
<span class="na">volumes</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">/var/run/docker.sock:/var/run/docker.sock:ro</span>
<span class="pi">-</span> <span class="s">./proxy/certs:/etc/nginx/certs:rw</span>
<span class="na">collabora</span><span class="pi">:</span>
<span class="na">image</span><span class="pi">:</span> <span class="s">collabora/code</span>
<span class="na">container_name</span><span class="pi">:</span> <span class="s">collabora</span>
<span class="na">expose</span><span class="pi">:</span>
<span class="pi">-</span> <span class="m">9980</span>
<span class="na">cap_add</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">MKNOD</span>
<span class="na">environment</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">domain=NEXTCLOUD_DOMAIN</span>
<span class="pi">-</span> <span class="s">VIRTUAL_HOST=COLLABORA_DOMAIN</span>
<span class="pi">-</span> <span class="s">VIRTUAL_NETWORK=proxy-net</span>
<span class="pi">-</span> <span class="s">VIRTUAL_PORT=9980</span>
<span class="pi">-</span> <span class="s">VIRTUAL_PROTO=https</span>
<span class="pi">-</span> <span class="s">LETSENCRYPT_HOST=COLLABORA_DOMAIN</span>
<span class="pi">-</span> <span class="s">LETSENCRYPT_EMAIL=ADMIN_EMAIL</span>
<span class="na">networks</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">proxy-net</span>
<span class="na">networks</span><span class="pi">:</span>
<span class="na">proxy-net</span><span class="pi">:</span>
<span class="na">external</span><span class="pi">:</span>
<span class="na">name</span><span class="pi">:</span> <span class="s">proxy-net</span>
</code></pre></div></div>
<p>You’ll just need to adapt <code class="language-plaintext highlighter-rouge">NEXTCLOUD_DOMAIN</code>, <code class="language-plaintext highlighter-rouge">COLLABORA_DOMAIN</code> (2x) and <code class="language-plaintext highlighter-rouge">ADMIN_EMAIL</code>. Also the Network, that connects the services (=collabora) with the proxy is declared as external, so you’ll have to create it by running</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>docker network create <span class="nt">--driver</span> bridged proxy-net
</code></pre></div></div>
<p>before starting the containers with</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>docker-compose up
</code></pre></div></div>
<blockquote>
<p>Note: if you also want to have access to Collabora’s admin interface at https://COLLABORA_DOMAIN/loleaflet/dist/admin/admin.html also provide <code class="language-plaintext highlighter-rouge">username</code> and <code class="language-plaintext highlighter-rouge">password</code> in the <em>environment</em> section for the Collabora container.</p>
</blockquote>
<h3 id="link-nextcloud-and-collabora">Link Nextcloud and Collabora</h3>
<p>All that’s left to do is enter the domain of your collabora service into the settings of your nextcloud collabora app and start editing documents right inside your nextcloud!
<img src="/assets/collabora-for-nextcloud/settings.png" alt="Nextcloud - Collabora Settings" /></p>
<p><a href="http://creativecommons.org/licenses/by-sa/4.0/">CC-BY-SA-4.0</a></p>Prelude I started writing this blog post in a first attempt a couple days ago and it got way too verbose and lengthy, going into detail about the specifics of my setup and the reasoning behind it. So here’s a short version. Goal Adding Collabora to an existing Nextcloud instance to enable online collaborative document editing right inside the Nextcloud webinterface. Step by Step Prerequisites I’m assuming you’ve got Nextcloud installed somewhere already (publically reachable, with it’s own domain or subdomain) and also installed the Collabora app (it’s listed as one of the official apps in the Office & Text section of Nextcloud’s list of apps). You also need a server for running Collabora, that has docker and docker-compose installed and a domain or subdomain pointing to it. If both, Nextcloud and Collabora, will be running on the same server, take a closer look at the jwilder/nginx-proxy, that I’ll be using anyhow. It should be no problem having both running behind the proxy on the same server, accessible through different subdomains.Using one Yubikey with two (not completely) separate PGP-Identities2019-03-16T00:00:00+00:002019-03-16T00:00:00+00:00https://blog.dwagenk.com/2019/03/one-yubikey-multiple-PGP-identities<h2 id="prelude">Prelude</h2>
<p>I planned writing some posts about embedded systems development. Well, I’ve been too busy to actually get started with that and although I’ve got some half-baked posts somewhere on my hard drive, this blog has been empty up to now.
This post is a little out of the embedded systems programming scope I had in mind for this blog, but I’ve spent quite some time on getting it all together and felt the urge to properly document it. Leastwise to give some guidance to my future self about the setup of my PGP keys.</p>
<h2 id="the-desired-pgp-setup">The desired PGP setup</h2>
<p>I’m not a heavy duty PGP user, but I’ve had keys setup for my e-mail addresses for about 5 years now. I’m using one e-mail address for personal, but more or less formal correspondence, and one, that is based on a pseudonym of mine and that I use for correspondence with just a few people.</p>
<p>Let’s use an (partly realistic, partly fictional) example:</p>
<p>Suppose I’m a person called <em>Daniel Wagenknecht</em> and mainly I’m using the e-mail address <em>dwagenk@mailprovider.org</em>. I’m also enthusiastic about comics and in those circles I’m known as <em>comicfreak_dan</em>, using the e-mail address <em>comicfreak_dan@comics.com</em>. Since I’m privacy aware I’m using PGP whenever it is not tooooo inconvenient. Putting both of my e-mail addresses into the same PGP identity results in something like this:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>sec rsa4096/0x73E8D88F 2019-03-12 <span class="o">[</span>SCE]
uid <span class="o">[</span>ultimate] Daniel Wagenknecht <dwagenk@mailprovider.org>
uid <span class="o">[</span>ultimate] comicfreak_dan <comicfreak_dan@comics.com>
</code></pre></div></div>
<p>It looks wrong, using it is like introducing myself as <em>Daniel Wagenknecht a.k.a comicfreak_dan</em> when I’m opening a bank account or meeting my doctor. I don’t want to link my pseudonym to all my formal email correspondence.</p>
<p>That’s why, when I first got started using PGP, I decided to use two identities instead. One is linked to my main, formal e-mail address and a second one goes with my pseudonymous email address.</p>
<p>The handling was easy, I just needed both private keys on my computer and for convenience set the same password for both of them.</p>
<h3 id="in-comes-the-yubikey">In comes the Yubikey</h3>
<p>Now after having had a Yubikey for a couple months and using it for 2-Factor-Authentication with some services, I also wanted to use it for PGP encryption and signing. But a single Yubikey has exactly one PGP subkey slot for each:</p>
<ul>
<li>encryption</li>
<li>signing</li>
<li>Authentication</li>
</ul>
<p>So to keep the two PGP identities separate I could either use two Yubikeys, or transition only one PGP identity to the Yubikey and keep the other one in the <em>traditional</em> way. Both solutions didn’t really satisfy me.</p>
<p>My idea now was to use two identities, meaning two separate PGP masterkeys, but both holding the same subkeys for encryption, signing and authentication. So the desired key structure looks similar to this:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>sec rsa4096/0x73E8D88F 2019-03-12 <span class="o">[</span>C]
uid <span class="o">[</span>ultimate] Daniel Wagenknecht <dwagenk@mailprovider.org>
ssb rsa4096/0x87D67E3D 2019-03-12 <span class="o">[</span>S] <span class="o">[</span>expires: 2020-03-11]
ssb rsa4096/0xA4C5604D 2019-03-12 <span class="o">[</span>A] <span class="o">[</span>expires: 2020-03-11]
ssb rsa4096/0xC26C348F 2019-03-12 <span class="o">[</span>E] <span class="o">[</span>expires: 2020-03-11]
sec rsa4096/0xDD6AA744 2019-03-12 <span class="o">[</span>C]
uid <span class="o">[</span>ultimate] comicfreak_dan <comicfreak_dan@comics.com>
ssb rsa4096/0x87D67E3D 2019-03-12 <span class="o">[</span>S] <span class="o">[</span>expires: 2020-03-11]
ssb rsa4096/0xA4C5604D 2019-03-12 <span class="o">[</span>A] <span class="o">[</span>expires: 2020-03-11]
ssb rsa4096/0xC26C348F 2019-03-12 <span class="o">[</span>E] <span class="o">[</span>expires: 2020-03-11]
</code></pre></div></div>
<p>Of course there is a drawback. As you can see above, someone who has both my public keys will be able to reach the conclusion, that both keys belong to me. If both keys are present on the keyservers, then searching for one of the subkey ids there will also list both master keys and make it pretty obvious, that both belong to me.
I’m not living a dual life, so avoiding any link between the two of my identities by all means is not necessary and I’m fine with the solution. Of curse this could very well differ for you. Maybe you are a superhero and need to separate your identities? Maybe you’re just living in a country where you’re better off staying incognito as a political activist (always a good idea e.g. looking at Turkey right now or at the repression linked to the G20 summit in Hamburg, Germany about 1 <sup>1</sup>/<sub>2</sub> years ago).</p>
<p>Also I don’t know, if people having more experience with PGP than I do, will disapprove with this concept, because different masterkeys sharing the same subkeys is something not intended by PGP.</p>
<h2 id="howto-use-the-same-subkeys-on-a-yubikey-with-different-master-keys">HowTo: use the same subkeys on a Yubikey with different master keys</h2>
<p>There’s a couple of tutorials online, on how to set up PGP / GPG / Yubikey to work together. I’d recommend reading https://github.com/drduh/YubiKey-Guide. Here’s just a quick summary:</p>
<ol>
<li>Use an offline computer for the key generation and keeping your master key. If you want to play around and do a dry-run, use a temporary GNUPGHOME (<code class="language-plaintext highlighter-rouge">export GNUPGHOME=$(mktemp -d) ; echo $GNUPGHOME</code>)</li>
<li>Ensure you’ve got a secure GPG configuration following best practices, both on your regular and your computer for key generation</li>
<li>create a new key with only the certification capability</li>
<li>add subkeys for signing, encryption and authentication</li>
<li>backup your keys to a secure location</li>
<li>configure your Yubikey</li>
<li>move the subkeys to your Yubikey</li>
<li>copy, import and trust your public key on your devices</li>
<li>use it!</li>
</ol>
<p>I’ll not be going into detail about the common parts of this procedure, but focus on the <em>two masterkeys sharing identical subkeys</em> part. GPG comes with a way to import existing keys, using something called the keygrip, that allows us to copy the subkeys from one master key to another.
On my first try using the subkeys with the second master key I failed, because the imported subkeys had different fingerprints then the original ones despite being made up of the same private key material. The cause: fingerprints hash not only the private key, but also it’s timestamp. So to have identical fingerprints for our subkeys we’ll have to create a timewarp later…</p>
<h3 id="now-lets-get-started">Now let’s get started!</h3>
<ol>
<li>Teach your computer how to create a timewarp, install <code class="language-plaintext highlighter-rouge">faketime</code>.
<ul>
<li>Ubuntu: <code class="language-plaintext highlighter-rouge">sudo apt-get install faketime</code></li>
<li>Arch: <code class="language-plaintext highlighter-rouge">sudo pacman -Sy libfaketime</code></li>
<li>Mac: faketime should be available, too</li>
<li>Windows: I’ve seen a tool called RunAsDate, that could work for our purposes</li>
</ul>
<p>Reach out to me, if you’ve got more information about how to make it work on Windows/Mac and I can include it here.</p>
</li>
<li>create a master key with only the certification capability
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="nv">NAME</span><span class="o">=</span><span class="s2">"Daniel Wagenknecht"</span>
<span class="nv">MAIL</span><span class="o">=</span><span class="s2">"dwagenk@mailprovider.org"</span>
<span class="nv">COMMENT</span><span class="o">=</span><span class="s2">""</span>
<span class="o">{</span>
<span class="nb">echo </span>8 <span class="c"># RSA (set your own capabilities)</span>
<span class="nb">echo </span>S <span class="c"># Toggle off signing</span>
<span class="nb">echo </span>E <span class="c"># Toggle off encryption</span>
<span class="nb">echo </span>Q <span class="c"># Finished with setting own capabilities</span>
<span class="nb">echo </span>4096 <span class="c"># Key length</span>
<span class="nb">echo </span>0 <span class="c"># Key does not expire</span>
<span class="nb">echo </span>y <span class="c"># is this correct?</span>
<span class="nb">echo</span> <span class="nv">$NAME</span> <span class="c"># Real Name</span>
<span class="nb">echo</span> <span class="nv">$MAIL</span> <span class="c"># Email address</span>
<span class="nb">echo</span> <span class="nv">$COMMENT</span> <span class="c"># Comment</span>
<span class="nb">echo </span>O <span class="c"># Okay</span>
<span class="o">}</span> | gpg2 <span class="nt">--command-fd</span><span class="o">=</span>0 <span class="nt">--status-fd</span><span class="o">=</span>1 <span class="nt">--expert</span> <span class="nt">--full-generate-key</span>
<span class="c"># Save the key-id of the generated key in environment for later use</span>
<span class="nv">KEYID</span><span class="o">=</span>id_of_your_just_created_key
</code></pre></div> </div>
<p>Since I like to program, meaning I’d rather spent hours on automating a task, than repeatedly answering interactive prompts, I’m using bash syntax and some flags for GPG to pass in input to GPG. Check out <a href="https://serverfault.com/a/858164">this</a> answer on StackExchange/ServerFault for details.</p>
<p>You should just need to adjust the variables at the top and copy the whole block into a terminal. The password prompts will still be interactive.</p>
<p>If you want to know what’s actually happening, do the steps interactively by running <code class="language-plaintext highlighter-rouge">gpg2 --expert --full-generate-key</code> and using the echo statements above as guidance on what to select.</p>
</li>
<li>Create the second master key
As I mentioned above we need to bend time a little and GPG will complain, if it notices what we’re doing. Creating the second master key before the subkeys will prevent that.
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="nv">NAME</span><span class="o">=</span><span class="s2">"comicfreak_dan"</span>
<span class="nv">MAIL</span><span class="o">=</span><span class="s2">"comicfreak_dan@comics.com"</span>
<span class="nv">COMMENT</span><span class="o">=</span><span class="s2">""</span>
<span class="o">{</span>
<span class="nb">echo </span>8 <span class="c"># RSA (set your own capabilities)</span>
<span class="nb">echo </span>S <span class="c"># Toggle off signing</span>
<span class="nb">echo </span>E <span class="c"># Toggle off encryption</span>
<span class="nb">echo </span>Q <span class="c"># Finished with setting own capabilities</span>
<span class="nb">echo </span>4096 <span class="c"># Key length</span>
<span class="nb">echo </span>0 <span class="c"># key does not expire</span>
<span class="nb">echo </span>y <span class="c"># is this correct?</span>
<span class="nb">echo</span> <span class="nv">$NAME</span> <span class="c"># Real Name</span>
<span class="nb">echo</span> <span class="nv">$MAIL</span> <span class="c"># Email address</span>
<span class="nb">echo</span> <span class="nv">$COMMENT</span> <span class="c"># Comment</span>
<span class="nb">echo </span>O <span class="c"># Okay</span>
<span class="o">}</span> | gpg2 <span class="nt">--command-fd</span><span class="o">=</span>0 <span class="nt">--status-fd</span><span class="o">=</span>1 <span class="nt">--expert</span> <span class="nt">--full-generate-key</span>
<span class="c"># Save the key-id of the generated key in environment for later use</span>
<span class="nv">KEYID2</span><span class="o">=</span>id_of_your_just_created_key
</code></pre></div> </div>
</li>
<li>add subkeys to the first master key
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="c"># Signing</span>
<span class="o">{</span>
<span class="nb">echo </span>addkey
<span class="nb">echo </span>8 <span class="c"># RSA (set your own capabilities)</span>
<span class="nb">echo </span>E <span class="c"># Toggle off encryption</span>
<span class="nb">echo </span>Q <span class="c"># Finished with setting own capabilities</span>
<span class="nb">echo </span>4096 <span class="c"># Key length</span>
<span class="nb">echo </span>1y <span class="c"># Key is valid for 1 year</span>
<span class="nb">echo </span>y <span class="c"># is this correct?</span>
<span class="nb">echo </span>y <span class="c"># really create?</span>
<span class="nb">echo </span>save
<span class="o">}</span> | gpg2 <span class="nt">--command-fd</span><span class="o">=</span>0 <span class="nt">--status-fd</span><span class="o">=</span>1 <span class="nt">--expert</span> <span class="nt">--edit-key</span> <span class="nv">$KEYID</span>
<span class="c"># Encryption</span>
<span class="o">{</span>
<span class="nb">echo </span>addkey
<span class="nb">echo </span>8 <span class="c"># RSA (set your own capabilities)</span>
<span class="nb">echo </span>S <span class="c"># Toggle off signing</span>
<span class="nb">echo </span>Q <span class="c"># Finished with setting own capabilities</span>
<span class="nb">echo </span>4096 <span class="c"># Key length</span>
<span class="nb">echo </span>1y <span class="c"># Key is valid for 1 year</span>
<span class="nb">echo </span>y <span class="c"># is this correct?</span>
<span class="nb">echo </span>y <span class="c"># really create?</span>
<span class="nb">echo </span>save
<span class="o">}</span> | gpg2 <span class="nt">--command-fd</span><span class="o">=</span>0 <span class="nt">--status-fd</span><span class="o">=</span>1 <span class="nt">--expert</span> <span class="nt">--edit-key</span> <span class="nv">$KEYID</span>
<span class="c"># Authentication</span>
<span class="o">{</span>
<span class="nb">echo </span>addkey
<span class="nb">echo </span>8 <span class="c"># RSA (set your own capabilities)</span>
<span class="nb">echo </span>S <span class="c"># Toggle off signing</span>
<span class="nb">echo </span>E <span class="c"># Toggle off encryption</span>
<span class="nb">echo </span>A <span class="c"># Toggle on Authentication</span>
<span class="nb">echo </span>Q <span class="c"># Finished with setting own capabilities</span>
<span class="nb">echo </span>4096 <span class="c"># Key length</span>
<span class="nb">echo </span>1y <span class="c"># Key is valid for 1 year</span>
<span class="nb">echo </span>y <span class="c"># is this correct?</span>
<span class="nb">echo </span>y <span class="c"># really create?</span>
<span class="nb">echo </span>save
<span class="o">}</span> | gpg2 <span class="nt">--command-fd</span><span class="o">=</span>0 <span class="nt">--status-fd</span><span class="o">=</span>1 <span class="nt">--expert</span> <span class="nt">--edit-key</span> <span class="nv">$KEYID</span>
</code></pre></div> </div>
</li>
<li>
<p>collect all the necessary data about the subkeys</p>
<p>We need each subkeys so called <em>keygrip</em> and it’s <em>creation timestamp</em>.</p>
<p>Enter
<code class="language-plaintext highlighter-rouge">gpg2 --list-secret-keys --with-keygrip --with-colons</code> and extract the needed information. See the highlighted parts in the output.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[...]
sec:u:4096:1:AE299F4373E8D88F:1552427224:::u:::cC:::+:::23::0:
fpr:::::::::4C5A569630B7FFEEF29A811BAE299F4373E8D88F:
grp:::::::::C020F1AC4B24B51C71E2481547DA6B926E8E7001:
uid:u::::1552427224::B2FC8BCA72AC2CD7656861523F4B4F6752E42F3C::Daniel Wagenknecht <dwagenk@mailprovider.org>::::::::::0:
ssb:u:4096:1:074D014F87D67E3D:<b><i>1552427749</i></b>:1583963749:::::<b><i>s</i></b>:::+:::23:
fpr:::::::::814F2B6C98BCE6F780187695074D014F87D67E3D:
grp:::::::::<b><i>0F3765677327A472B3FE52F01B93A22A5E202D13</i></b>::
ssb:u:4096:1:571C4F86C26C348F:<b><i>1552427813</i></b>:1583963813:::::<b><i>e</i></b>:::+:::23:
fpr:::::::::AB9F307FA0D9A648ED165193571C4F86C26C348F:
grp:::::::::<b><i>008FF4813D7897EE8DD33A27FDA533F9AF71C7DC</i></b>:
ssb:u:4096:1:1428372BA4C5604D:<b><i>1552427780</i></b>:1583963780:::::<b><i>a</i></b>:::+:::23:
fpr:::::::::3B1D7D423AABBDF6EB62B38D1428372BA4C5604D:
grp:::::::::<b><i>E703AA6F7A9FCA9F91E1FFC4F38B30081021CFA4:</i></b>
[...]</code></pre></div></div>
<p>So in my case the values are</p>
<table>
<thead>
<tr>
<th>Subkey</th>
<th>Timestamp</th>
<th>Keygrip</th>
</tr>
</thead>
<tbody>
<tr>
<td>s = sign</td>
<td>1552427749</td>
<td>0F3765677327A472B3FE52F01B93A22A5E202D13</td>
</tr>
<tr>
<td>e = encrypt</td>
<td>1552427813</td>
<td>008FF4813D7897EE8DD33A27FDA533F9AF71C7DC</td>
</tr>
<tr>
<td>a = authenticate</td>
<td>1552427780</td>
<td>E703AA6F7A9FCA9F91E1FFC4F38B30081021CFA4</td>
</tr>
</tbody>
</table>
</li>
<li>
<p>import the subkeys to the second master key</p>
<p>I haven’t found a way to automatically feed the keygrips to gpg, you’ll need to copy them into the interactive prompts.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="c"># Signing</span>
<span class="nv">TIMESTAMP_S</span><span class="o">=</span>1552427749
<span class="o">{</span>
<span class="nb">echo </span>addkey
<span class="nb">echo </span>13 <span class="c"># Existing key </span>
<span class="nb">echo </span>E <span class="c"># Toggle off encryption</span>
<span class="nb">echo </span>Q <span class="c"># Finished with setting own capabilities</span>
<span class="nb">echo </span>1y <span class="c"># key is valid for 1 year</span>
<span class="nb">echo </span>y <span class="c"># is this correct?</span>
<span class="nb">echo </span>y <span class="c"># really create?</span>
<span class="nb">echo </span>save
<span class="o">}</span> | faketime <span class="s2">"</span><span class="si">$(</span><span class="nb">date</span> <span class="nt">-d</span>@<span class="nv">$TIMESTAMP_S</span><span class="si">)</span><span class="s2">"</span> gpg2 <span class="nt">--command-fd</span><span class="o">=</span>0 <span class="nt">--status-fd</span><span class="o">=</span>1 <span class="nt">--expert</span> <span class="nt">--edit-key</span> <span class="nv">$KEYID2</span>
</code></pre></div> </div>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="c"># Encryption</span>
<span class="nv">TIMESTAMP_E</span><span class="o">=</span>1552427813
<span class="o">{</span>
<span class="nb">echo </span>addkey
<span class="nb">echo </span>13 <span class="c"># RSA (set your own capabilities)</span>
<span class="nb">echo </span>S <span class="c"># Toggle off signing</span>
<span class="nb">echo </span>Q <span class="c"># Finished with setting own capabilities</span>
<span class="nb">echo </span>1y <span class="c"># key is valid for 1 year</span>
<span class="nb">echo </span>y <span class="c"># is this correct?</span>
<span class="nb">echo </span>y <span class="c"># really create?</span>
<span class="nb">echo </span>save
<span class="o">}</span> | faketime <span class="s2">"</span><span class="si">$(</span><span class="nb">date</span> <span class="nt">-d</span>@<span class="nv">$TIMESTAMP_E</span><span class="si">)</span><span class="s2">"</span> gpg2 <span class="nt">--command-fd</span><span class="o">=</span>0 <span class="nt">--status-fd</span><span class="o">=</span>1 <span class="nt">--expert</span> <span class="nt">--edit-key</span> <span class="nv">$KEYID2</span>
</code></pre></div> </div>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="c"># Authentication</span>
<span class="nv">TIMESTAMP_A</span><span class="o">=</span>1552427780
<span class="o">{</span>
<span class="nb">echo </span>addkey
<span class="nb">echo </span>13 <span class="c"># RSA (set your own capabilities)</span>
<span class="nb">echo </span>S <span class="c"># Toggle off signing</span>
<span class="nb">echo </span>E <span class="c"># Toggle off encryption</span>
<span class="nb">echo </span>A <span class="c"># Toggle on Authentication</span>
<span class="nb">echo </span>Q <span class="c"># Finished with setting own capabilities</span>
<span class="nb">echo </span>1y <span class="c"># key is valid for 1 year</span>
<span class="nb">echo </span>y <span class="c"># is this correct?</span>
<span class="nb">echo </span>y <span class="c"># really create?</span>
<span class="nb">echo </span>save
<span class="o">}</span> | faketime <span class="s2">"</span><span class="si">$(</span><span class="nb">date</span> <span class="nt">-d</span>@<span class="nv">$TIMESTAMP_A</span><span class="si">)</span><span class="s2">"</span> gpg2 <span class="nt">--command-fd</span><span class="o">=</span>0 <span class="nt">--status-fd</span><span class="o">=</span>1 <span class="nt">--expert</span> <span class="nt">--edit-key</span> <span class="nv">$KEYID2</span>
</code></pre></div> </div>
<p>Now we’re done with the <em>two masterkeys sharing identical subkeys</em> part. You can go on with the regular procedure already outlined.</p>
</li>
<li>backup your keys to a secure location</li>
<li>configure your Yubikey. You can set a pub key url, and the corresponding key will work out of the box on any machine that has a working PGP / GPG setup and can reach that url.</li>
<li>move the subkeys to the Yubikey (only once, although they are associated with 2 master keys)</li>
<li>copy, import and trust your public key on your devices</li>
<li>use it!</li>
</ol>
<h3 id="a-note-on-openkeychain-for-android">A note on Openkeychain for Android</h3>
<p>On my smartphone I’m using K9Mail and Openkeychain for my encrypted mails. Openkeychain has support for the Yubikey via NFC (or USB-C, depending on which device you got), so there’s no worries about jeopardizing my private keys when reading and writing encrypted mail on there. When configuring Openkeychain with a Yubikey, it will read the subkeys ids from the Yubikey and search for a matching master key in your keychain and online. It works fine for the first masterkey, but will not import the second one without using a trick.</p>
<ol>
<li>open Openkeychain and go to <code class="language-plaintext highlighter-rouge">manage private keys</code> (name could be slightly off, I’ve got it installed in german)</li>
<li>select <code class="language-plaintext highlighter-rouge">use security-token</code></li>
<li>if it finds one of your public keys (through keyserver or the url saved on the Yubikey), this key should be ready to use. If not provide it as a file.</li>
<li>test the first key on the device</li>
<li>backup the first key</li>
<li>delete the first key</li>
<li>repeat steps 1.-3. but make sure it doesn’t find the first public key, but the second (e.g. set your phone to airplane mode and provide the second public key via file)</li>
<li>test the second key on the device</li>
<li>restore the first key</li>
<li>it should work with both keys now</li>
</ol>
<p>With this setup I’ve experienced both, GPG / Enigmail and Openkeychain mentioning the wrong user id / masterkey when trying to decrypt something, but decryption works anyway. The same could happen on any device, that has both of my public keys in the keychain, so someone who’s got both my public keys imported might get the notice, that an email is signed by <em>dwagenk@mailprovider.org</em>, whilst it was sent by <em>comicfreak_dan@comics.com</em>. This is unfortunate, but I guess with a setup like this it’s better to really separate the use of the keys: each of your correspondents should only have one of your public keys.</p>
<p>Thanks for reading. Send me an e-mail or open an issue in the <a href="https://github.com/DWagenk/blog.dwagenk.me">repo</a> if you’ve got any comments on this post. I’m interested in some discussion about my PGP setup!</p>
<p><a href="http://creativecommons.org/licenses/by-sa/4.0/">CC-BY-SA-4.0</a></p>Prelude I planned writing some posts about embedded systems development. Well, I’ve been too busy to actually get started with that and although I’ve got some half-baked posts somewhere on my hard drive, this blog has been empty up to now. This post is a little out of the embedded systems programming scope I had in mind for this blog, but I’ve spent quite some time on getting it all together and felt the urge to properly document it. Leastwise to give some guidance to my future self about the setup of my PGP keys.