<?xml version="1.0" encoding="utf-8"?><feed xmlns="http://www.w3.org/2005/Atom" ><generator uri="https://jekyllrb.com/" version="3.10.0">Jekyll</generator><link href="https://arifordsham.com/feed.xml" rel="self" type="application/atom+xml" /><link href="https://arifordsham.com/" rel="alternate" type="text/html" /><updated>2026-01-07T10:09:12+00:00</updated><id>https://arifordsham.com/feed.xml</id><title type="html">Adventures in Software</title><subtitle>Ari Fordsham&apos;s blog</subtitle><entry><title type="html">Rust is more permissive than C, but C has better concurrency support</title><link href="https://arifordsham.com/rust-is-permissive/" rel="alternate" type="text/html" title="Rust is more permissive than C, but C has better concurrency support" /><published>2026-01-07T00:00:00+00:00</published><updated>2026-01-07T00:00:00+00:00</updated><id>https://arifordsham.com/rust-is-permissive</id><content type="html" xml:base="https://arifordsham.com/rust-is-permissive/"><![CDATA[<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>f() + g();
</code></pre></div></div>

<p>What happens if <code class="language-plaintext highlighter-rouge">f</code> and <code class="language-plaintext highlighter-rouge">g</code> touch the same bit of state?</p>

<ul>
  <li>
    <p>In <strong>Rust</strong>, this is perfectly valid, and <code class="language-plaintext highlighter-rouge">f</code> runs strictly before <code class="language-plaintext highlighter-rouge">g</code>.</p>
  </li>
  <li>
    <p>In <strong>C</strong>, this construct is not legal - it is ‘undefined behaviour.’ Your compiler
won’t (can’t) disallow it, but it is allowed to assume that it never happens.</p>
  </li>
</ul>

<p>This means that <em>theoretically</em>, a <strong>C</strong> compiler is allowed to automatically parallelize
this code, but in <strong>Rust</strong>, we must wait for <code class="language-plaintext highlighter-rouge">f</code> to complete before <code class="language-plaintext highlighter-rouge">g</code> is able to run.</p>

<p>Enjoy the rest of your day.</p>]]></content><author><name></name></author><summary type="html"><![CDATA[f() + g();]]></summary></entry><entry><title type="html">How to run Linux commands without installing</title><link href="https://arifordsham.com/run-linux-commands-without-installing/" rel="alternate" type="text/html" title="How to run Linux commands without installing" /><published>2023-07-31T00:00:00+00:00</published><updated>2023-07-31T00:00:00+00:00</updated><id>https://arifordsham.com/run-linux-commands-without-installing</id><content type="html" xml:base="https://arifordsham.com/run-linux-commands-without-installing/"><![CDATA[<p>Maybe you want to try out a Linux command, or use it as a one-off. If you’re 
like me, it makes you vaguely uneasy to have your machine littered with software
you forgot why you installed. Wouldn’t it be nice if there was some way of 
running commands, from the Internet, without having them leaving residue on your
machine? Sounds impossible, no?</p>

<p>(Github Copilt is sitting on my shoulder, and it suggested Docker. Nice try, but
not quite. Besides for the trouble of getting it set up, what if I want easy 
access to my local files or hardware? Try again, wise guy.)</p>

<p>I’m going to show you a solution using <a href="https://nixos.org/">Nix</a>. Nix is a part 
package manager, part
build system, that has tremendous power, but also suffers currently from 
tremendous <a href="https://github.com/AriFordsham/nix-gripes">usability issues</a>. I want
to show you how to use its power, while avoiding the frustrating bits.</p>

<p>To run a command without installing:</p>

<ul>
  <li>
    <p>First you need to <a href="https://nixos.org/download.html">install Nix</a>. Basically,
you execute this:</p>

    <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>sh &lt;<span class="o">(</span>curl <span class="nt">-L</span> https://nixos.org/nix/install<span class="o">)</span> <span class="nt">--daemon</span>
</code></pre></div>    </div>
  </li>
</ul>

<p>It will call <code class="language-plaintext highlighter-rouge">sudo</code> to install, unfortunately, but it can mainly be used 
afterward without superuser privileges.</p>

<ul>
  <li>
    <p>You will need to restart your shell for Nix to work properly.</p>
  </li>
  <li>
    <p>Next, configure Nix for all the latest features:</p>

    <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span><span class="nb">echo</span> <span class="s2">"experimental-features = nix-command flakes"</span> | <span class="nb">sudo tee</span> <span class="nt">-a</span> /etc/nix/nix.conf
</code></pre></div>    </div>
  </li>
  <li>
    <p>Now, find the name of the Nix package and command you want using the 
<a href="https://search.nixos.org/packages">Nix package search</a>. You may know the name 
of the command, or you can find a list of commands available by package in the
package details section.</p>
  </li>
  <li>
    <p>If the command is the same as the package name (or the package is configured 
with a default command), you can run it like this:</p>

    <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>nix run nixpkgs#&lt;package&gt; <span class="nt">--</span> &lt;args&gt;
</code></pre></div>    </div>

    <p>If you need to run a different command from the package, the invocation is:</p>

    <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>nix shell nixpkgs#&lt;package&gt; <span class="nt">-c</span> &lt;<span class="nb">command</span><span class="o">&gt;</span> <span class="nt">--</span> &lt;args&gt;
</code></pre></div>    </div>

    <p>Nix will download the package and all its dependencies to an isolated location
on your system, then execute the command!</p>

    <p>If you’re going to invoke a command from a package more than once, you can 
start a bash shell with a package loaded:</p>

    <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>nix shell nixpkgs#&lt;package&gt; ...
</code></pre></div>    </div>

    <p>You can specify more than one package. The commands from the package then 
become avalable until you exit the Nix shell with <code class="language-plaintext highlighter-rouge">Ctrl-D</code>.</p>
  </li>
</ul>

<h3 id="what-happens-to-the-downloaded-packages-afterward">What happens to the downloaded packages afterward?</h3>

<p>Nix downloads all it’s packages and all their dependencies right down to the 
Linux kernel API to an isolated location on your system - <code class="language-plaintext highlighter-rouge">/nix/store</code>. Nix will
completely ignore any programs or libraries you have installed on your system. 
It then keeps them around in case you want to use them again. (This is not as 
useful as it sounds, because a second invocation of the same package is likely 
to redownload if there is a new version of the package <em>or any of its 
dependencies</em>.)</p>

<p>So a disadvantage of using Nix is that it can consume a lot of disk space. You
can free up space by clearing cached packages with:</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>nix store gc
</code></pre></div></div>

<p>You should configure Nix to use less space by:</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span><span class="nb">echo</span> <span class="s2">"auto-optimise-store = true"</span> | <span class="nb">sudo tee</span> <span class="nt">-a</span> /etc/nix/nix.conf
</code></pre></div></div>

<h2 id="addendum-installing-packages-without-root-or-building">Addendum: Installing packages without root (or building)</h2>

<p>Often, you may want to give a Linux user without sudo privileges the ability to
install packages. None of the popular package managers provide this.</p>

<p>There is a reason for this. Installing packages into the system default location
needs root privileges, of course. So instead, you want to install to a directory
within the user’s home directory. Trouble is, Linux programs typically have many
paths, such as dynamic libraries, hardcoded into their binaries. Therefore, the 
package managers that distribute prebuilt binaries can only install into 
predetermined locations. Gentoo, which builds all packages from source, indeed
allows restricted users to install packages with a custom ‘prefix.’</p>

<p>The next option is to build everything you need fro source, but this is 
time-consuming and error-prone.</p>

<p>With Nix, restricted users can install packages into their own user profiles.
Nix installs all packages int its ‘store’, and considers it safe to allow restricted
users to do so, as the store is read-only. You can install packages with:</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>nix profile <span class="nb">install </span>nixpkgs#&lt;package&gt;
</code></pre></div></div>

<p>This creates a garbage collection ‘root’, so this package won’t go away if you 
run <code class="language-plaintext highlighter-rouge">nix store gc</code>. You can remove it with:</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>nix profile remove packages.x86_64-linux.&lt;package&gt;
</code></pre></div></div>

<p>These are some of the powerful and cool things you can do with Nix. Nix has the
ability to do much more, but it’s not for the faint of heart.</p>]]></content><author><name></name></author><summary type="html"><![CDATA[Maybe you want to try out a Linux command, or use it as a one-off. If you’re like me, it makes you vaguely uneasy to have your machine littered with software you forgot why you installed. Wouldn’t it be nice if there was some way of running commands, from the Internet, without having them leaving residue on your machine? Sounds impossible, no?]]></summary></entry><entry><title type="html">How To Set Up Your WFH Office</title><link href="https://arifordsham.com/how-to-set-up-your-wfh-office/" rel="alternate" type="text/html" title="How To Set Up Your WFH Office" /><published>2023-04-23T00:00:00+00:00</published><updated>2023-04-23T00:00:00+00:00</updated><id>https://arifordsham.com/how-to-set-up-your-wfh-office</id><content type="html" xml:base="https://arifordsham.com/how-to-set-up-your-wfh-office/"><![CDATA[<p>Remote working brings a new level of work flexibility, but it does mean a lot of
communication by video call.</p>

<p>Video calling introduces challenges that are not present in trad face-to-face
communication. Research shows that a huge proportion of the information exchange
in a verbal conversation is non-verbal, including body language. Video calling
either eliminates or reduces the fidelity of much of this ‘out of band’ 
information. If you talk with your hands, like I do, this can be left out of the
video entirely if you’re not careful. Bad quality audio and video also reduces
the reliability of audible and facial expressions.</p>

<p>We are wired to be most comfortable and at ease in a conversation when the other
person (hereafter “you” for the sake of chattiness) is within a certain distance
range, and again, when we can see their head and shoulders and maybe a certain 
amount of their chest.</p>

<p>Eye contact is an important component of the nonverbal part of a conversation.
This is difficult to achieve over video. You are looking at me through my camera,
and I am looking at you on my screen. Depending on the setup, you may:</p>

<ul>
  <li>
    <p>Be looking down at my forehead or the top of my head (typical webcam placement, exacerbated on big
screens)</p>
  </li>
  <li>
    <p>Be looking up at me, at my chin (Some laptops have the webcam down by the laptop hinge)</p>
  </li>
  <li>
    <p>Be looking at me from the side (When I attempt to position the camera at eye level).
This looks particularly rude because it looks like I am looking away from you 
and at my computer screen, a sure sign of lack of attention in a face-to-face
conversation.</p>
  </li>
</ul>

<p>All in all, my wishlist for a video call setup is:</p>

<ol>
  <li>
    <p>Video quality is as good as possible</p>
  </li>
  <li>
    <p>Eye contact occurs naturally and is perceived correctly</p>
  </li>
  <li>
    <p>You can see enough of me that feels natural, and you can see my hands move</p>
  </li>
  <li>
    <p>You can see as little as possible of my desk and the room behind me</p>

    <p>My room is a mess, and even if it wasn’t, it will be an unnecessary distraction
  during the call.</p>
  </li>
  <li>
    <p>You can see whether I am looking at you or at my computer screen</p>
  </li>
  <li>
    <p>Your picture is a comfortable size and distance from me.</p>

    <p>This is subjective, some people may in fact want to conceal this, but I personally prefer
  an authentic and transparent approach.</p>
  </li>
  <li>
    <p>There is no curving distortion of the picture</p>
  </li>
</ol>

<p>What people typically get is a low-to-middle quality webcam, often their laptop’s
built-in one, mounted on top of their screen. This leads to a close-up headshot,
far from ideal. My idea of a better setup follows.</p>

<p><em>Disclaimer: I haven’t had the space or the equipment to try everything in this
setup, so some of it remains untested and theoretical.</em></p>

<p>It would seem that points (3) and (4) are a contradiction, but in fact they are
not. The key lies in the “camera angle”, often known as “field of view” or “focal length”<sup id="fnref:1" role="doc-noteref"><a href="#fn:1" class="footnote" rel="footnote">1</a></sup>. To
cut through the jargon, when the camera is close by, you can only show a lot of
foreground by showing a lot of background. In order to get the most context, your
camera’s lens may even need to be set up to give a curved distortion.</p>

<p>However, if the camera is placed at a distance and zoomed in, you can far more easily show 
context of your body while only showing the part of the room directly behind you.</p>

<p>As such, my ideal setup would be: A high-quality webcam (point 1) with a narrow focal length (“zoomed in”),
mounted on top of a large-screen TV, placed slightly off to the side and a few feet across
the room.</p>

<p>This provides the best balance between context of my body (3) and the room (4), without needing distortion (7). It
also puts you in a place which is comfortable for me (6). Because of the distance, the angle between camera and display is reduced, leading to more natural eye contact (2).</p>

<p>In fact, if you’ve ever seen a newsroom, they use cameras set right back from the newscaster, giving that head/shoulder/body shot that feels most natural to viewers.</p>

<p>The next step then, is choosing a camera. We’re looking for a camera with a narrow focal angle.
Unfortunately, narrow focal-angle webcams are too niche to exist. The next best we
can do is adjustable focal length - also known as “optical zoom” (digital zoom just crops the image, not changing the focal length), but it needs to go narrower than standard. The webcam market seems to focus on broader angle, getting more context from a traditional closeup mount point - leading to the curved distortion I mentioned earlier.</p>

<p>There is a <a href="https://amzn.to/3H7islz">cheap webcam</a> on Amazon with a manually adjustable focal length, but
I didn’t want to take a risk on quality. Luckily, The Logitech Brio models have
a field of view that adjusts down to 65 degrees, which works very well in practice.
There are two models available - The <a href="https://amzn.to/3L2V3mr">BRIO 500</a> with 1080p resolution and the <a href="https://amzn.to/440cIUS">4K Pro</a> (also known as the BRIO Stream) with 4K and other advanced features. I have the BRIO Stream and am very happy with it. (Be aware that may video-conferencing platforms limit the streaming resolution, canceling some of the benefits of a better camera.) Once you have your camera positioned as you like, you change the focal length in the Logitech LogiTune app.</p>

<p>One thing I’d love to understand better is lighting. As far as I understand so far,
The right lighting makes a big difference to video quality, and the best place for the lights
is behind the camera, with as much lighting as possible. Obviously right behind doesn’t work,
so probably you need lights up and to the sides. This jives with what I’ve seen of photo studios.</p>

<div class="footnotes" role="doc-endnotes">
  <ol>
    <li id="fn:1" role="doc-endnote">
      <p>I credit a chapter in Randall Munroe’s book “How To” for helping me
  understand this concept. <a href="#fnref:1" class="reversefootnote" role="doc-backlink">&#8617;</a></p>
    </li>
  </ol>
</div>]]></content><author><name></name></author><summary type="html"><![CDATA[Remote working brings a new level of work flexibility, but it does mean a lot of communication by video call.]]></summary></entry><entry><title type="html">What Can I Refactor For You Today?</title><link href="https://arifordsham.com/what-can-i-refactor/" rel="alternate" type="text/html" title="What Can I Refactor For You Today?" /><published>2021-07-01T00:00:00+00:00</published><updated>2021-07-01T00:00:00+00:00</updated><id>https://arifordsham.com/what%20can-i-refactor</id><content type="html" xml:base="https://arifordsham.com/what-can-i-refactor/"><![CDATA[<p>As developers, we love building things. Haskell gives us great opportunities to channel our creative drives into producing libraries and tools that are (hopefully) both intellectually elegant and functionally useful. But there’s more to building software than just creating.</p>

<p>Unfortunately, software often needs maintenance, or other changes after the initial write. On older projects, library or GHC changes mean code might no longer compile. A project may have been architected without accounting for ways it may need to change in the future, and extending it is now a struggle. In the rush to get something completed, an engineer may be left with an implementation that he cannot be confident is free of bugs, or is simply not satisfyingly tidy or elegant.</p>

<p>That’s why I’d like to offer my services, as a Haskell consultant, to maintain and refactor existing code, either yours or someone else’s. If any of the following apply to you, I’d love to know if I can be of service.</p>

<ul>
  <li>
    <p>You’re interested in using a Haskell tool or library, but it -  or one of its dependencies - has ‘bit-rotted’ - it won’t build on current GHC/<code class="language-plaintext highlighter-rouge">base</code>.</p>
  </li>
  <li>
    <p>You’ve got a project you’d like to bring up to date with more recent GHC features or library idioms.</p>
  </li>
  <li>
    <p>You have a project using library <em>x</em>, and you’d like to port it to use library <em>y</em> instead.</p>
  </li>
  <li>
    <p>You wish the internals of your project were easier to reason with than they are.</p>
  </li>
  <li>
    <p>You’d like to untangle pure code from <code class="language-plaintext highlighter-rouge">IO</code>, or invert dependencies to make your program more extensible.</p>
  </li>
  <li>
    <p>You’d like to introduce typeclasses to generalize and extend your code.</p>
  </li>
  <li>
    <p>You’d like to lift some of your program’s semantics from runtime errors and <code class="language-plaintext highlighter-rouge">Maybe</code>s into the type system.</p>
  </li>
  <li>
    <p>You’d like to move your project, or a library you are using, over to the <code class="language-plaintext highlighter-rouge">stack</code> build system for more dependable dependency tracking and reproducible builds.</p>
  </li>
  <li>
    <p>You’d like to see a library (possibly somebody else’s) added to Stackage for the CI quality assurance that provides.</p>
  </li>
</ul>

<p>I’m especially excited to see how I can leverage Facebook’s <a href="https://github.com/facebookincubator/retrie"><code class="language-plaintext highlighter-rouge">retrie</code></a> tool to automate changes over sizable codebases.</p>

<p>Dependable refactoring needs test coverage to make sure you’re not breaking anything. Types help with this, but they are not the whole answer - There’s still a need for tests. I can also help with writing or expanding test suites.</p>

<p>Of course, I’m also available to assist you along your Haskell journey in other ways - whether through development or training/tutoring.</p>

<p>If you think I might be able to be of assistance with your project, you can <a href="https://docs.google.com/document/d/1hSYvXiOq99diyL_6Ulj0ENJ2neXxc4NZ7SrUaMMEw0A/edit?usp=sharing">read my CV</a>, and be in touch by email to <a href="mailto:arifordsham@gmail.com">arifordsham@gmail.com</a>.</p>

<p>Ari Fordsham</p>]]></content><author><name></name></author><summary type="html"><![CDATA[As developers, we love building things. Haskell gives us great opportunities to channel our creative drives into producing libraries and tools that are (hopefully) both intellectually elegant and functionally useful. But there’s more to building software than just creating.]]></summary></entry><entry><title type="html">What I don’t like about Github Copilot</title><link href="https://arifordsham.com/what-i-dont-like-about-copilot/" rel="alternate" type="text/html" title="What I don’t like about Github Copilot" /><published>2021-06-01T00:00:00+00:00</published><updated>2021-06-01T00:00:00+00:00</updated><id>https://arifordsham.com/what-i-dont-like-about-copilot</id><content type="html" xml:base="https://arifordsham.com/what-i-dont-like-about-copilot/"><![CDATA[<p>DISCLAIMER: I haven’t had a chance to use Copilot - I’m on the waitlist, like everyone else. I’m writing based on the impressions I’ve got from the <a href="https://copilot.github.com/">quite nice website</a>, the underlying <a href="https://beta.openai.com/">OpenAI Codex</a> technology, and my own rudimentary ideas of how machine learning works.</p>

<p>Microsoft, the owners of Github, annouced that a new project – Github Copilot – entered Technical Preview on the 29th June 2021. It’s an extension to the Visual Studio Code editor that analyzes your code and suggests completions – even multi-line functions – based on a machine learning algorithm, that was trained on all of Github. From the online examples and testimonials, this sounds like a a Big Deal.</p>

<p>You are now invited to <a href="https://github.com/features/copilot/signup">join the waitlist</a> and who knows? you may be one of the lucky ones who gets to give it a spin. Spoiler: Github have <a href="https://copilot.github.com/#faqs">said</a> “we are offering GitHub Copilot to a limited number of testers for free” and “if the technical preview is successful, our plan is to build a commercial version of GitHub Copilot in the future.” The production version almost certainly won’t be available free, and probably won’t be cheap. If it’s as good as it promises, it could become a staple of commercial programming shops, but indie developers like me who don’t <em>need</em> this might not be able to justify it. Oh well, just saying.</p>

<p>I think Copilot does have genuine promise, and the possibility of making meaningful improvements to the process of coding. But from what I’ve seen – and this makes sense – Copilot does not <em>create</em>. All it does is attempt to piece together an ad-hoc description of the programmer’s meaning from information embedded in code, comments and names into hopefully correct executable code. Your informal description probably needs to contain barely less detail than the code itself. In this way, Copilot transforms and sharpens, rather than innovates<sup id="fnref:2" role="doc-noteref"><a href="#fn:2" class="footnote" rel="footnote">1</a></sup>.</p>

<p>There’s a joke going around: after Copilot comes Pilot, and then I’ll have to get a new job (if there are any). Jokes aside, getting from Copilot to Pilot is WAY bigger than getting from nothing to Copilot - it’s not even the same kind of thing.</p>

<p>But here’s the problem. The most fundamental principle of software development is DRY – Don’t Repeat Yourself. Repeating the same code in two places is a code smell - you should find some way of writing it only once, such as putting it in a function.
The formal name for avoiding repetition is <em>abstraction</em>.</p>

<p>Now, in order for a model like Copilot to learn, it needs a lot of redundancy in it’s dataset. The success of Copilot hinges on the fact that in the code we write today, there is still loads of duplication – we’re not nearly there yet. And the real solution to redundancy is getting better at abstraction – finding and using new abstraction mechanisms, and learning how to apply abstraction more widely and effectively<sup id="fnref:1" role="doc-noteref"><a href="#fn:1" class="footnote" rel="footnote">2</a></sup>.</p>

<p>Copilot’s breakthrough party trick is the ability to analyze natural language in comments and function and variable names to come up with suggestions. However, many of the examples on the website show that Copilot can – and does – suggesti multi-line code snippets, the kind of thing I look at and think: that shouldn’t be pasted in, it should be abstracted away. By doing this, Copilot is locking in a duplicative coding style. At the same time as Copilot is doing your work for you, it’s also taking away the pain of being verbose and repetitive and thereby encouraging copy-paste-style coding. I worry how this could hold back progress on searching for abstractions. Abstraction is more than saving developer time – it can even slow you down up front. Rather, it helps make code more readable and maintainable (by moving irrelevant detail away), it can even make code more correct and performant (by spending the time getting it right once), but above all – <em>it adds to our understanding of software development</em> – it lets us recognise these pattens, even give them a name, and thereby guides us to think higher level. Blindly pasting code from Copilot doesn’t do any of these things.</p>

<p>Of course, on the flip side, Copilot might encourage developers to write comments, which is a good thing. Then again, they probably won’t delete or trim the comments after they write(?) the code, and therefore the comments will repeat what the code says, which is bad<sup id="fnref:3" role="doc-noteref"><a href="#fn:3" class="footnote" rel="footnote">3</a></sup>.</p>

<p>Can we use AI to find abstractions? Very possibly, and that would be amazing. but it will be a lot harder than merely analyzing and parroting code, and no one the size of Microsoft/Github is even trying.</p>

<p>I’m not knocking Co-pilot – it definitely seems to be, rather than a gimmick, an amazing product that has the potential to transform the way we code, and especially help developers bridge the gap from junior to proficient. I just hope the activity of programming will not take a step back as a result.</p>
<div class="footnotes" role="doc-endnotes">
  <ol>
    <li id="fn:2" role="doc-endnote">
      <p>This observation, by the way, can be generalized to every machine learning (ML) project I have seen to date – very powerful but blunt generalization tools that don’t really have more than ‘one level’ of insight (whatever that means.) Can AI do more? Very possibly, but I haven’t yet seen a single proof-of-concept. <a href="#fnref:2" class="reversefootnote" role="doc-backlink">&#8617;</a></p>
    </li>
    <li id="fn:1" role="doc-endnote">
      <p>At this point I’ll give a plug for my favorite technologies: pure functional programming languages, especially <a href="https://www.haskell.org/">Haskell</a>. Functional languages take abstraction very seriously – they have a bunch of powerful abstraction mechanisms, and their communities try very hard to discover new ways of encoding common patterns. Haskell programs use fewer lines of code to accomplish the same tasks. It would be very interesting to me to see if Copilot is less effective on Haskell because it has less redundancy. <a href="#fnref:1" class="reversefootnote" role="doc-backlink">&#8617;</a></p>
    </li>
    <li id="fn:3" role="doc-endnote">
      <p>See <a href="https://learning.oreilly.com/library/view/the-practice-of/9780133133448/ch01.html#ch01lev1sec6">Chapter 1 of The Practice of Programming</a> for why. <a href="#fnref:3" class="reversefootnote" role="doc-backlink">&#8617;</a></p>
    </li>
  </ol>
</div>]]></content><author><name></name></author><summary type="html"><![CDATA[DISCLAIMER: I haven’t had a chance to use Copilot - I’m on the waitlist, like everyone else. I’m writing based on the impressions I’ve got from the quite nice website, the underlying OpenAI Codex technology, and my own rudimentary ideas of how machine learning works.]]></summary></entry><entry><title type="html">How I reduced my Haskell CI time by 84%</title><link href="https://arifordsham.com/how-i-reduced-my-ci-time-by-84-percent/" rel="alternate" type="text/html" title="How I reduced my Haskell CI time by 84%" /><published>2021-05-14T00:00:00+00:00</published><updated>2021-05-14T00:00:00+00:00</updated><id>https://arifordsham.com/how-i-reduced-my-ci-time-by-84-percent</id><content type="html" xml:base="https://arifordsham.com/how-i-reduced-my-ci-time-by-84-percent/"><![CDATA[<p>I know how confounded I was by CI before I got into it, and how straightforward it seems now, so I thought I’d write down my experiences for anyone following in my footsteps.</p>

<p>In this post, I will speak about how I implemented Gitlab Continuous integration (CI), and then sped up execution time dramatically by setting the right options. My project is Haskell-centric, but many of the takeaways can be applied  to any language.</p>

<p>I use Michael Snoyman’s excellent <a href="https://docs.haskellstack.org/en/stable/README/">stack</a> build tool for my Haskell projects. Amongst many features that generally improve my quality of life as a developer, stack tries to guarantee <em>reproducible builds</em> - controlling as many variables as possible, to ensure build behaviour is the same - reproducible - between builds, even if I change the configuration of my machine, or even move the project to a different machine.</p>

<p>To achieve this, stack downloads all project dependencies into the project directory. But it goes further - it downloads a fixed version of GHC, the Haskell compiler, and stashes it away in a special location. This ensures every version uses a specific version of the build toolchain, and updating my system GHC install  will never break a project due to a subtle change in compiler behaviour, for example.</p>

<p>I have a shell script I use that defines what I consider a full working build for my project, stored at <code class="language-plaintext highlighter-rouge">ci/build.sh</code>:</p>

<script src="https://gist.github.com/AriFordsham/54862f3eea9314af29da0e12d9331648.js?file=ci-build.sh"></script>

<ul>
  <li>Run the stack build command to build the project (based on the poject’s <code class="language-plaintext highlighter-rouge">package.yaml</code> file.)</li>
  <li>Run all test suites</li>
  <li>Generate a <a href="https://wiki.haskell.org/Haskell_program_coverage">test coverage</a> report</li>
  <li>Generate <a href="https://www.haskell.org/haddock/">haddock</a> documentation</li>
  <li>The <code class="language-plaintext highlighter-rouge">$@</code> passes command-line options from the script through to <code class="language-plaintext highlighter-rouge">stack build</code> - useful for one-off build scenarios.</li>
</ul>

<p>My project is hosted on Gitlab, so I wanted to get GitLab’s excellent CI to run and validate this script on every push.</p>

<p>Initial setup was dead straightforward (once I knew how!): I created a <code class="language-plaintext highlighter-rouge">.gitlab-ci.yml</code> file in the root of my project as follows:</p>

<script src="https://gist.github.com/AriFordsham/54862f3eea9314af29da0e12d9331648.js?file=.gitlab-ci-initial.yml"></script>

<ul>
  <li>Define which Docker image to use. I’m using <a href="https://hub.docker.com/r/migamake/stack-build-image">stack-build-image</a> from <a href="https://migamake.com/">Migamake</a>, which provides stack preinstalled, for <a href="https://github.com/commercialhaskell/lts-haskell#lts-haskell-version-your-ecosystem">LTS Haskell</a> version 17.</li>
  <li>Define the <code class="language-plaintext highlighter-rouge">stack</code> job, which runs the script <code class="language-plaintext highlighter-rouge">ci/build.sh</code>.</li>
</ul>

<p>Job done! On every push, Gitlab runs the build script in a Docker container. Since <code class="language-plaintext highlighter-rouge">stack</code> returns an error code if any test suite fails, I get a big red cross next to my commit on GitLab if my project fails to build or run correctly.</p>

<p>Here comes the issue: after writing that script, ever CI run took in excess of 33 minutes. Not that this is really a problem in any way: My CI script also runs in a pre-commit hook<sup id="fnref:1" role="doc-noteref"><a href="#fn:1" class="footnote" rel="footnote">1</a></sup> (which runs in under a minute on my development machine), so I never push without knowing CI passes. It’s just nice to not have to wait long for that satisfying green tick, and with all those bitcoin miners around, It feels better not to be unneccesarily squandering the planet’s resources.</p>

<p>A quick look at the build log shows the problem. As mentioned, stack downloads the GHC compiler and all dependencies. Since Docker does not store any state between runs, the build system has loads of downloading and compiling to do before it can even start building and testing the project itself.</p>

<p>That’s just unnecessary duplication of work - the dependencies don’t change between runs. On my local machine, stack keeps a cache of already-built libraries, so rebuilds are near-instant. However, to ensure reproducibility, all Docker runs start with a clean slate (besides whatever is in the container itself,) so this cache is not kept.</p>

<p>Gitlab’s caching feature comes to the rescue. You can tell Gitlab that changes to certain directories won’t affect the build correctness of your project, so Gitlab will go ahead and preserve those between runs. I added the following to my <code class="language-plaintext highlighter-rouge">.gitlab-ci.yml</code>:</p>

<script src="https://gist.github.com/AriFordsham/54862f3eea9314af29da0e12d9331648.js?file=.gitlab-ci-cache.yml"></script>

<p>This sets up Gitlab’s cache for the <code class="language-plaintext highlighter-rouge">.stack-work/</code> directory, where stack keeps library dependencies<sup id="fnref:2" role="doc-noteref"><a href="#fn:2" class="footnote" rel="footnote">2</a></sup>. Now, at the end of every run, Gitlab zips up the <code class="language-plaintext highlighter-rouge">.stack-work</code> directory, and uploads it to a cloud bucket. Then, next time, before it begins running your CI scripts, it downloads and unzips it into your project directory.</p>

<p>This should allow stack to now find its cache from last time, and should enable zippy-fast builds, like I get on my local machine.</p>

<p>However, pushing the build script (twice; once to create the initial cache, and then again to see how it’s used) gives only a negligible improvement in CI run time. A glance at the build log seems to show that the cache is working correctly. So what is going on?</p>

<p>stack caches project-specific artifacts in <code class="language-plaintext highlighter-rouge">.stack-work</code>, under the project directory. However, the bulk of its local cache, including the GHC build chain and many libraries, is stored by default in <code class="language-plaintext highlighter-rouge">.stack</code> under the user home directory. This enables the sharing of the cache between projects.</p>

<p>Now there’s an additional complication: While Docker containers provide a home directory for projects to use, Gitlab cache only works for directories under the project directory. So I also need to set the <code class="language-plaintext highlighter-rouge">$STACK-ROOT</code> environment variable, to tell stack to store its build chain where the cache can see it.</p>

<p>My final <code class="language-plaintext highlighter-rouge">.gitlab-ci.yml</code> looks like this:</p>

<script src="https://gist.github.com/AriFordsham/54862f3eea9314af29da0e12d9331648.js?file=.gitlab-ci.yml"></script>

<p>Success! After generating the cache, run time has dropped from 32 minutes to 7 minutes.</p>

<p>A look at the build log shows that this time is dominated by downloading and uploading the cache zip. Even adding <code class="language-plaintext highlighter-rouge">--fast</code> to <code class="language-plaintext highlighter-rouge">ci/build.sh</code>, which tells stack to do unoptimized builds, made negligble difference, so I concluded there’s no more simple optimization opportunities.</p>

<p>Since I initially set up CI, my project has grown and added dependencies. So I ran the latest version, with and without the cache enabled.</p>

<p>A regular run now takes just under eight minutes, but without the cache, it takes almost 50 minutes, or an 84% reduction in run time.</p>

<p>CI looks daunting, but it simple enough once you invest the time to learn how to do what you want, andif you do it right (and don’t overcomplicate!) you can significantly improve and smooth your workflow.</p>

<div class="footnotes" role="doc-endnotes">
  <ol>
    <li id="fn:1" role="doc-endnote">
      <p>Here’s how you do a precommit hook:</p>

      <p>Your git repo has a hidden directory, <code class="language-plaintext highlighter-rouge">.git</code>, where git stores it’s bookkeeping. Under that directory is a directory called <code class="language-plaintext highlighter-rouge">hooks</code> (By default, it contains a set of <code class="language-plaintext highlighter-rouge">.sample</code> files showing what hooks are available). I created a file there called <code class="language-plaintext highlighter-rouge">pre-commit</code>:</p>

      <script src="https://gist.github.com/AriFordsham/54862f3eea9314af29da0e12d9331648.js?file=.git-hooks-pre-commit"></script>

      <p>On Linux, you’ll generally need to set this as executable by running <code class="language-plaintext highlighter-rouge">chmod +x .git/hooks/pre-commit</code>.</p>

      <p>This will now run before every commit. As with CI, an error code from the script will cause the commit to fail. You can override this with <code class="language-plaintext highlighter-rouge">git commit --force</code>. <a href="#fnref:1" class="reversefootnote" role="doc-backlink">&#8617;</a></p>
    </li>
    <li id="fn:2" role="doc-endnote">
      <p>The key field is required: it allows you to use different caches for different scenarios, such as branches, by setting the key to an environment variable. I trust stack to always use the right library versions from the <code class="language-plaintext highlighter-rouge">package.yaml</code>, and so the more sharing the better. Therefore I just use an arbitrary contant string so all scenarios share a single cache. <a href="#fnref:2" class="reversefootnote" role="doc-backlink">&#8617;</a></p>
    </li>
  </ol>
</div>]]></content><author><name></name></author><summary type="html"><![CDATA[I know how confounded I was by CI before I got into it, and how straightforward it seems now, so I thought I’d write down my experiences for anyone following in my footsteps.]]></summary></entry><entry><title type="html">Haskell - Doomed to Succeed?</title><link href="https://arifordsham.com/haskell-doomed-to-succeed/" rel="alternate" type="text/html" title="Haskell - Doomed to Succeed?" /><published>2020-12-16T00:00:00+00:00</published><updated>2020-12-16T00:00:00+00:00</updated><id>https://arifordsham.com/haskell-doomed-to-succeed</id><content type="html" xml:base="https://arifordsham.com/haskell-doomed-to-succeed/"><![CDATA[<p>The unofficial motto of Haskell, the predominant lazy functional language, has long been:</p>

<blockquote>
  <p><em>“Avoid success at all costs.”</em></p>
</blockquote>

<p>This is attributed to Simon L. Peyton-Jones (SPJ), the main architect of the Haskell language and the GHC compiler, and mentioned in the <a href="https://www.microsoft.com/en-us/research/wp-content/uploads/2016/07/HaskellRetrospective.ppt">slides</a> of a <a href="https://www.microsoft.com/en-us/research/publication/wearing-hair-shirt-retrospective-haskell-2003/">talk</a> he gave at the POPL conference in 2003.</p>

<p>This seems an odd statement. What’s wrong with a bit of success?</p>

<p>The exact meaning of this statement is controversial. Simon Marlow, another major Haskell figure, <a href="https://twitter.com/simonmar/status/246335257677271040">quotes SPJ</a> that this statement (expression?) should not be bracketed <code class="language-plaintext highlighter-rouge">(avoid success) at all costs</code>, i.e. do all you can to ensure success doesn’t happen, but rather <code class="language-plaintext highlighter-rouge">avoid (success at all costs)</code> - success may be a nice idea, but avoid falling in to the trap of focusing all the community’s efforts to attaining it.</p>

<p>But isn’t success the goal of any project?</p>

<p>The answer can be found in an <a href="https://www.aosabook.org/en/ghc.html">article</a> written by the aformentioned Marlow and Peyton-Jones:</p>

<blockquote>
  <p>[T]he ultimate goal for us, the main developers of GHC, is to produce research rather than code. We consider developing GHC to be an essential prerequisite: the artifacts of research are fed back into GHC, so that GHC can then be used as the basis for further research that builds on these previous ideas.</p>
</blockquote>

<p>In other words, the success referred to is <em>commercial</em> success, or wide user adoption. The stated goal of GHC (and the Haskell language) is to better understand the principles of functional programming, and the mathematical and logical principles that underly them.</p>

<p>A wide user base, using Haskell in important projects, poses a responsibility to the Haskell designers. Users have needs, and can push the project in directions irrelevant to the core research goals, as the authors go on to state:</p>

<blockquote>
  <p>[A] great deal of effort is put into ensuring that [GHC] can be relied on for production use. There has often been some tension between these two seemingly contradictory goals, but by and large we have found a path that is satisfactory both from the research and the production-use angles.</p>
</blockquote>

<p>They certainly seem to have been successful: GHC is widely considered a robust and reliable compiler, and the GHC developers take the quality of their product very seriously, despite their very limited resources.</p>

<p>So while SPJ claims to have been saying “don’t allow efforts to make the platform appealing to production users take away from Haskell’s key principles,” It definitely has been understood in practice to mean “Let’s try not to draw attention to ourselves, because too many users will prevent us doing what we want to do.”</p>

<p>Until now.</p>

<p>In his POPL talk, SPJ quotes Anthony Hoare as saying:</p>

<blockquote>
  <p><em>“I fear that Haskell is doomed to succeed.”</em></p>
</blockquote>

<p>Haskell’s advantages - a strong mathematical basis, facilities for abstraction and de-duplication, <a href="https://arifordsham.com/is-haskell-fast/">excellent performance compared to other high-level languages</a>, good correctness guarantees, ease of concurrency, but most of all, <em>enabling a better way to think about code</em>, have increasingly lead to its <a href="https://wiki.haskell.org/Haskell_in_industry">widespread adoption in production code</a>. Haskell is contending with success despite itself.</p>

<p>This shift in the makeup of the Haskell community certainly has led to a shift in focus. At the Haskell eXchange conference in November 2020, SPJ <a href="https://youtu.be/MEmRarBL9kw">announced</a> the <a href="https://haskell.foundation/">Haskell Foundation</a>, explicitly “dedicated to broadening the adoption of Haskell” as well as “supporting its ecosystem of tools, libraries, education, and research.”</p>

<p>So while the original architects of Haskell may find this unintended success unwanted, it is certainly a vindication of their vision: remaining faithful to strong mathematical principles and resisting calls for premature pragmatism can transform the things we do.</p>

<h3 id="update">UPDATE</h3>

<p>Simon Peyton-Jones has graciously commented on this post - you can read his remarks <a href="https://discourse.haskell.org/t/new-blog-post-haskell-doomed-to-succeed/1662/2">here</a>.</p>

<p>In short, he reiterates that wide production adoption was never an anti-goal of Haskell, but rather not “making fundamental compromises of core principles [of Haskell] in pursuit of short-term production goals.”</p>]]></content><author><name></name></author><summary type="html"><![CDATA[The unofficial motto of Haskell, the predominant lazy functional language, has long been:]]></summary></entry><entry><title type="html">Is Haskell fast?</title><link href="https://arifordsham.com/is-haskell-fast/" rel="alternate" type="text/html" title="Is Haskell fast?" /><published>2020-11-24T00:00:00+00:00</published><updated>2020-11-24T00:00:00+00:00</updated><id>https://arifordsham.com/is-haskell-fast</id><content type="html" xml:base="https://arifordsham.com/is-haskell-fast/"><![CDATA[<p>If you try to research this question, you might come up with confusing and contradictory answers, strongly-stated opinions, and loads of technical jargon.</p>

<p>The answer depends which perspective the questioner is coming from, and what assumptions, expectations and preconceived notions they bring with them.</p>

<h2 id="correctly-written-haskell-is-fast-for-a-high-level-language">Correctly written Haskell is fast for a high-level language</h2>

<p>Haskell is a <em>high-level language</em> - a tool used to write programs with a goal of avoiding the programmer having to specify implementation details, as far as practical. This is typically the most productive way of writing most software, where performance must be ‘adequate’ rather than optimum.</p>

<p>In this space, developers typically reach for <em>dynamically typed, interpreted</em> languages such as JavaScript and Python. This kind of language design has an inherent overhead, because there is no opportunity to eliminate runtime work ahead-of-time, and in particular the program has to shuffle around and bookkeep type information.</p>

<p>Haskell, on the other hand, is <em>type-erased</em> and <em>compiled</em>. The GHC compiler does an excellent job of optimizing Haskell code to make it run much faster than it otherwise might. So as long as the developer is aware of the necessary caveats and pitfalls (more on those later), Haskell programs will run dramatically faster than their HLL counterparts.</p>

<h2 id="straightforward-haskell-will-be-slower-than-c">Straightforward Haskell will be slower than C</h2>

<p>It remains a fact that code written in clear, idiomatic Haskell will (at present) be slower than code written in a low-level language such as C, C++ or Rust, which would typically be used for performance-critical code. GHC cannot yet optimize all the overhead from the Haskell abstractions, and will lose out to code that runs close to the metal, typically with a slowdown of around 50% to 4x.</p>

<p>This leaves Haskell in a bit of a tight spot when it comes to performance: it competes well against other high level languages in a domain where performance isn’t that important, but can’t quite keep up with the incumbents in the race for ultimate speed.</p>

<p>However, if someone is using Haskell for other reasons, to write more concise, maintainable code that will be easier to be confident in its correctness, they will get decent performance and responsiveness ‘for free!’</p>

<h2 id="you-can-write-performance-competitive-haskell">You can write performance-competitive Haskell</h2>

<p>So what to do with performance-critical code? One option is to write it in a low-level language, possibly calling in from Haskell through the Foreign Function Interface. In doing so, the programmer accepts the sacrifice of expressivity and safety, and must factor for the overhead of FFI and marshalling.</p>

<p>But there is another way. With a strong understanding of the GHC execution model, there is a set of techniques that can be used to write Haskell code that compiles to run at least as fast as equivalent C programs. This takes quite a bit of expertise to do, and it won’t be quite as tidy as idiomatic Haskell, but it should still be more readable (to the practiced eye) and maintainable than a highly-tuned C implementation.</p>

<p>A full rundown is beyond the scope of this post, but I’ll link to some resources:</p>

<ul>
  <li>
    <p><a href="https://donsbot.wordpress.com/2008/05/06/write-haskell-as-fast-as-c-exploiting-strictness-laziness-and-recursion/">Write Haskell as fast as C: exploiting strictness, laziness and recursion</a> (Written in 2008)</p>
  </li>
  <li>
    <p><a href="http://fixpt.de/blog/2017-12-04-strictness-analysis-part-1.html">All About Strictness Analysis</a> (How to take advantage of the latest compiler optimizations)</p>
  </li>
</ul>

<h2 id="watch-out-for-the-haskell-performance-traps">Watch out for the Haskell performance traps</h2>

<p>There are, however, some things that every production Haskell developer needs to be aware of. There is Haskell code that looks correct, but will run <em>way</em> slower, and use far more memory, than the worst you will come across in an interpreted language. The slowdown can run to several factors, with programs occasionally running out of stack space before they can get anything done!</p>

<p>This is due to a controversial design decision called <em>laziness</em>, compounded by, in hindsight, suboptimal library design and idiom choice, and needs to be understood in order to be able to guarantee consistent reasonable performance from your Haskell code.</p>

<p>I won’t go into all the details here, but in short, never use <code class="language-plaintext highlighter-rouge">foldl</code> (use <code class="language-plaintext highlighter-rouge">foldl'</code> instead), know how to use bang patterns, and read this article by Michael Snoyman:</p>

<ul>
  <li><a href="https://www.fpcomplete.com/blog/2017/09/all-about-strictness/">All about strictness</a></li>
</ul>

<h2 id="abstraction-doesnt-have-to-cost-it-can-even-pay">Abstraction doesn’t have to cost; it can even pay</h2>

<p>People seem to think there is inevitably a price to pay for writing high-level code. In fact the opposite is true. Theoretically speaking, high-level code is easier to optimize than low-level code. It simply provides more information to the compiler about the programmer’s intent.</p>

<p>As an illustration, let’s take the case of Rust. Rust is considered on par with C and C++ in terms of performance, despite being somewhat higher level. In spite of this, if you look through the <a href="https://github.com/rust-lang/rust/labels/I-slow">Rust issues on Github</a>, you’ll see many instances of the ‘I-slow’ tag, suggesting unexploited optimization opportunities. This suggests Rust is set to get faster yet.</p>

<p>How can this be? Surely everyone knows nothing beats hand-written C code?</p>

<p>I believe this to be a widely held myth. Optimization is more-or-less an exact science, and machines must eventually outperform humans. And ultimately, a perfect Haskell compiler will outperform a perfect optimizing C compiler.</p>

<p>Why is this?</p>

<ul>
  <li>
    <p>A C program prescribes precisely how a problem is to be solved. There is a limit how far the compiler can go in rearranging the code without breaking the language semantics. A high-level program describes the <em>problem</em>, and gives the compiler more freedom to choose the best implementation.</p>
  </li>
  <li>
    <p>Few programmers have a comprehensive knowledge of how best to optimize their programs. Compiler optimizations can be written once by a community of domain experts, and then applied widely.</p>
  </li>
  <li>
    <p>Even the most experienced C programmer must keep objectives other than sheer speed in mind while writing his code. The program must be able to be evaluated for correctness without losing his train of thought, and it must be extendable and maintainable, often by other people. In the source code, these are unavoidable trade-offs.</p>
  </li>
  <li>
    <p>Because of this, even programmers steeped in C performance technique must program defensively to ensure program correctness. A compiler can perform aggressive optimization that will make a programmer’s jaw drop, simply because it can keep track of far more program state at a time in order to gaurantee correctness. The compiler can then often build optimizations on other optimizations. Imagine trying to aggressively inline a program by hand.</p>
  </li>
</ul>

<h2 id="summary">Summary</h2>

<p>As long as it your code is written with an eye to a few common pitfalls, Haskell is a fast language compared to it’s closest competitors, although not quite fast enough to compete in the speed stakes. Heavily hand-optimized Haskell <em>can</em> be competitive, and we can expect compiler advances to further narrow the gap for more idiomatic code in coming years.</p>]]></content><author><name></name></author><summary type="html"><![CDATA[If you try to research this question, you might come up with confusing and contradictory answers, strongly-stated opinions, and loads of technical jargon.]]></summary></entry></feed>