Making a qBittorrent Module for NixOS

Tinix I -- Porting ESP-Hosted for the Luckfox Pico

I've had a Kindle Touch 5 (KT5) since they were released, around 2012. I've
used it on and off over the years, and it's still rock solid. I can't really
fault the device, it's the only piece of hardware that I've owned for 10+ years
that still works without issue.

Despite not having any issues, I've recently become interested in Jailbreaking
my device. The main motivation for this is gaining root access to the device
and installing [KOReader][1] which appears to be a much improved eink reader 
app. Jailbreaking my device should allow for better integration with my [home
server setup][2], without purchasing a new/different device.

## Finding the right jailbreak 
Most of the kindle jailbreaking info is on the [MobileRead forums][3]. The main
issue is figuring out which jailbreak to use. This depends both on the device
you are using and the installed firmware.

See [here][4] for identifying the device. Once you've identified the device,
I suggest checking out this [snapshots][5] page. It basically aggregates all
the kindle jailbreaks into a single location. Using both your device firmware
and device version, you should be able to identify the most suitable jailbreak.

Just search for that and you should be able to find an appropriate MobileRead
post with instructions.

## Jailbreaking
For my device and firmware I found the following to be most suitable:
- K5 JailBreak (5.0.x - 5.4.4.2)

This is available [here][6]. This is a neat little jailbreak, allowing me to root
the device without factory resetting.

I'll reproduce the instructions below for posterity:

---

Download the K5 JailBreak package from the Snapshots thread,
and unpack it. In here, you'll find a few files.
Unzip the contents of the kindle-5.4-jailbreak.zip archive
to the root directory of your Kindle.

Now, eject & unplug your Kindle, and go to
[HOME] -> [MENU] > Settings -> [MENU] > Update Your Kindle.
The updater won't actually ever run, this is normal.
After a few seconds, the words **** JAILBREAK **** will appear
at the bottom of your screen to confirm that the device is jailbroken.
You won't see this message again, it's simply a confirmation message.

And that's it, your Kindle is now ready to install custom hacks!

Since v1.11.N: If you need to use something MKK-powered immediately,
be aware that MKK will only be active on the next boot.
Installing another package through the Update mechanism will do the trick,
or you can manually trigger a device restart: > ([HOME] -> [MENU] >
Settings -> [MENU] > Restart).

---

I should note that the jailbreak worked. Now to install some new software.

## KUAL
Most of the the third-party kindle software relies on [KUAL][7]. This is an
app launcher for kindle, and is usually a requirement for any apps. Installing
this is straightforward. I've reproduced the instructions below.

---
... you may need to install MKK: if your device runs on FW 5.x, and you're
using the K5 JB >= 1.11.N (or the associated "hotfix"), you don't need to do
anything, MKK is bundled with your JB. If you're using an older device, you
will need to install MKK yourself 

Finally, you need to install KUAL itself, by putting the appropriate azw2
file somewhere in the documents/ directory tree of your device.
First, download the latest KUAL archive from the Snapshots thread and unpack it.
Then, if your device runs on FW 5.x, use the KUAL-KDK-2.0.azw2 file. If your
device is older, use the KUAL-KDK-1.0.azw2 file.

And that's it for the basics!

---
For my jailbreak, MKK came installed, meaning I can just copy the azw2 file to
the documents directory.

I did this, then tried to launch KUAL immediately and failed. Turns out I
needed to reboot my device to properly install the jailbreak. After rebooting,
KUAL launched as expected!

## KUAL Extensions!
KUAL isn't really much on it's own. Instead, it aggregrates extensions. Let's
start by installing the recommended extensions!

First I installed Helper and blocked OTA updates. Silent OTA updates might
break the device...

Next I installed MRPI.

## KOReader
To install KOReader, you can follow the instructions [here][8]. In short,
download the right release and unzip to the root directoty. I used the `kindle`
release.

I was initially a little bit underwhelmed by KOReader. I tried reading my
`.mobi` files and they were atrociously slow and poorly styled. Note that these
were automatically converted from the `.epub` source by Calibre. Given that
`.epub` support was part of my motivation for switching, I downloaded the same
book [Rhythm of War - Grant Sanderson] in epub format onto my Kindle. Wow.
The speed of epubs was outstanding and they maintained the correct formatting.

I customised the look and feel--line spacing, margins, alt status--downloaded
dictionaries, and looked through some of the menus. I'm really impressed. There
are some great features, like the reading statistics and plugins, that really
sell this as a major upgraded for an otherwise very old kindle.

## Conclusion
If you're on the fence definitely give jailbreaking a go. I highly recommend
KOReader. There are some quirks, like menu overlay issues on waking, but the
experience is much better overall than the stock kindle reader. You get
granular control over text rendering, rich format support, better reading
statistics and much more.

[1]:https://koreader.rocks/
[2]:https://www.samkwort.com/home-server-v3
[3]:https://www.mobileread.com/forums/forumdisplay.php?f=150
[4]:https://wiki.mobileread.com/wiki/Kindle_Serial_Numbers
[5]:https://www.mobileread.com/forums/showthread.php?t=225030
[6]:https://www.mobileread.com/forums/showthread.php?t=186645
[7]:https://www.mobileread.com/forums/showthread.php?t=203326
[8]:https://github.com/koreader/koreader/wiki/Installation-on-Kindle-devices


Jailbreaking a Kindle Touch 5

You may or may not have noticed that the website has changed. I built the old site,
*samkwort.com*, on a whim back in June/July 2020. I had pretty limited experience
with web development and wanted to get some familiarity with technologies that I
deemed to be quite accessible at the time. That old site was built using *Pelican*,
a static site generator implemented in Python. I remember thinking at the time that
I wanted to build a website without using any JavaScript. I was successful, though
I'm not sure why I thought that. Fast-forward three years and that old site started
to look pretty dated. Beyond that, I couldn't remember, nor did I document, how to
add new posts to the website. I figured I might as well rebuild it using modern web
technologies and take a deeper dive into the world of web development through
JavaScript.

## Goals
I wanted my new site to look better and be easier to create content for. I also
wanted to learn a bit about JavaScript. My experience had been limited to writing
a very small web scraper + JSON data server for an ESP32 project I've been working
on recently. I also wanted to learn a bit more about React; I was introduced to
React during an internship during the summer break--we were tasked with building
a custom front-end for generating reports based on Jenkins CI runs. I didn't
touch React, doing most of the backend Python-based data scraping. React seemed
pretty cool at that time. I actually sat across from a senior dev who happened
to be working on a front-end rewrite of an old Ember website. He kept saying that
any new JavaScript projects should really be written in TypeScript. I added that
to the list of things I wanted to learn.


## Implementation
I did some research into modern web frameworks and settled on *Next.js*; its
page-based routing and templating feature seemed quite intriguing, as well as
its relationship with React. It also seemed to be a bit of an industry standard.

### Layout
I started with the basic *create-next-app* template. I built the basic home-page,
creating custom *Header* and *Footer* components. Then, I created a *Layout*
component that would serve as the base for all my pages. Along the way, I picked
up a little bit of *Tailwind CSS*--styling my pages this way felt really simple.

The *Layout* element code looks like this:
```typescript
interface LayoutProps {
  children: React.ReactNode;
}

export default function Layout({ children }: LayoutProps): JSX.Element {
  return (
    <div className="mx-4 md:mx-auto md:max-w-6xl flex min-h-screen flex-col items-left p-4 md:p-24 font-mono">
      <Header/>
      <main>{children}</main>
      <Footer />
    </div>
  );
}
```
I really like the way React feels like HTML; the idea of custom components is
quite neat.

### PostList
I wanted the home-page to have a list of the most recent posts. To do this, I needed
to find a tool for parsing Markdown documents. After some research, I found the
*gray-matter* package. This parsed the Markdown into a JSON object containing data
and content fields; the data fields were generated using the YAML frontmatter of my
Markdown documents. 

I placed all my posts in a *posts* folder and wrote some code to generate a list of
Gray Matter objects. These were then sorted by the date field. I used
*getStaticProps* to ensure that the static *blogPosts* data was loaded before the
page was rendered.

Here's an excerpt of the code:

```typescript
interface HomeProps {
  postsData: Post[];
}

export async function getStaticProps(): Promise<{ props: HomeProps }> {
  const postsData = await getPostData();

  return {
    props: {
      postsData
    }
  };
}

export default function Home({ postsData }: HomeProps): JSX.Element {
  return (
    <Layout>
      <section>
        <h2 className="text-lg">Latest Posts</h2>
        <PostList postsData={postsData.slice(0, 5)} />
      </section>
    </Layout>
  );
}
```

### Dynamic routing with \[post\].tsx
*Next* offers dynamic routing, which makes it great for generating generic blog
post pages for any given Markdown document. Setting this up is fairly easy; the
[Next documentation][1] gives a pretty good example.

Once I had dynamic routes working and returning a basic hello-world type message,
I started working on parsing the Markdown blog posts into an abstracted
*BlogPost* component.

### Markdown formatting
The essence of the *BlogPost* component is Markdown to HTML conversion. I found
the perfect React component for this, [React Markdown][2]. This component takes
Markdown formatted text as its child and converts it to HTML.

I also wanted the option to display code and math in my blog posts. To set this
up, I followed the examples on the *React Markdown* GitHub page. The code
highlighting used the Syntax Highlighter package, while the math formatting
used remarkMath and rehypeKatex.

The final *BlogPost* component looks like this:

```typescript
<ReactMarkdown
  className="markdown"
  remarkPlugins={[remarkMath]}
  rehypePlugins={[rehypeKatex]}
  components={{
    code({ node, inline, className, children, ...props }) {
      const match = /language-(\w+)/.exec(className || '');
      return !inline && match ? (
        <SyntaxHighlighter
          {...props}
          style={irBlack}
          language={match[1]}
          PreTag="div"
        >
          {String(children).replace(/\n$/, '')}
        </SyntaxHighlighter>
      ) : (
        <code {...props} className={className}>
          {children}
        </code>
      );
    },
  }}
>
  {post.content}
</ReactMarkdown>
```

To style everything nicely, I used a dedicated stylesheet with Tailwind `@apply`
directives. I think it came together quite well. Also, here's some math:

$$
\begin{align*}
    D\int_\reals x^4\,u_{xx} \; dx &= -4\,D \int_\reals x^3\,u_{x} \; dx \\
                                  &= 12\,D \int_\reals x^2\,u \; dx
\end{align*}
$$


### Hosting
My domain *samkwort.com* was previously hosted by Google Domains, which was
recently sold to Squarespace. As such, I decided to migrate my domain to
Cloudflare after reading some very positive reviews online.

Like my previous site, hosting is done through GitHub Pages. I used the
[Deploy to GitHub Pages Action][4].

## Conclusion
JavaScript is weird. I definitely prefer coding in C and Python, but it was good
to branch out a little bit. I'm a big fan of the new site; making content is
super straightforward, and the design is a big upgrade.

[1]:https://nextjs.org/docs/pages/building-your-application/routing/dynamic-routes
[2]:https://github.com/remarkjs/react-markdown
[3]:https://github.com/vercel/next.js/tree/canary/examples/github-pages
[4]:https://github.com/marketplace/actions/deploy-to-github-pages







A New Website


As part of my current role at work, I'm required to automate the production of
many plots using results from PSCAD simulations. 
These plots are generated by `matplotlib`.
To streamline the plot reviewing process, we generally combine the plots into
a single PDF file. Currently, this process is incredibly slow, sometimes taking
minutes to generate a single PDF, with 50+ of pages.
As a learning exercise in Python benchmarking, and to improve performance, I
decided to undertake an investigation into this component of our simulation
workflow, hopefully removing the bottleneck.

## Current Practices
The current methodology for generating PDFs looks something like this:

1. Generate simulation data for each scenario, including plots.
2. Store folders names containing output plots in dictionary by 
   scenario number.
3. Iterate through output folders using dictionary and generate PDF
   for each scenario.

I narrowed down the performance bottleneck to step 3 above.
For this step, we have the following Python code:

```python
# Create unique pdf for each scenario
for key in pdf_groups.keys():
    pdf = FPDF(orientation='L', unit='mm', format='A4')
    
    # Loop through test in scenario, keeping track of first and last
    first_test = ""
    last_test = ""
    for test in pdf_groups[key]:
        if last_test == "":
            first_test = test

        for signal in SIGNALS_TO_CAPTURE:
            image = f"{out_folder}\\{test}\\{test}_{signal}.png"
            # Only add the graph to PDF if it exists.
            if (os.path.isfile(image)):
                pdf.add_page()
                pdf.image(image, x=0, y=20, w=300, h=150)
            else:
                print(f"File: {image} does not exist.", file=sys.stderr)

        last_test = test

pdf.output(out_folder + "\\" + first_test + "-" 
            + (re.search(r"[0-9]+",last_test)).group(0) + ".pdf", "F")
```

The code itself is $O(n\times m)$, where $n$ is the number of scenarios
and $m$ is the number of tests.

My intuition tells me that the bottleneck is somewhere in the `FPDF` package,
which is used here to embed the images in generated PDF output. We will
benchmark this entire code segment.

## Benchmarking
There are a few options for benchmarking the speed of code in Python. Some of
these are:

- [`timeit`][1]
- [`line_profiler`][2]
- [`pyperf`][3]

I'm opting to use `line_profiler` as this will give me a line by line breakdown
of my code's execution speed.

### Benchmarking Setup
To support the profiling process, we need to isolate this segment of code and
create a custom script for profiling. Here's what I ended up with.
```python
@profile
def benchmark_with_FPDF(tests, out_folder, in_folder):
    pdf = FPDF(orientation='L', unit='mm', format='A4')
    for test in tests:
        for signal in SIGNALS_TO_CAPTURE:
            image = in_folder + "\\" + test + "\\"+ test + "_" + signal + ".png"
            # Only add the graph to PDF if it exists.
            if (os.path.isfile(image)):
                pdf.add_page()
                pdf.image(image, x=0, y=20, w=300, h=150)
            else:
                print("File: " + image + " does not exist.")


    pdf.output(out_folder + "\\FPDF_benchmark.pdf", "F")
```

Note the use of the `@profile` decorator; `line_profiler` uses this to signal
which function to profile.

To profile the script with `line_profiler`, we just run:
```
kernprof -lv script_to_profile.py
```

### Results
Before using line_profiler, I ran a very crude test shown below.
```python
start_time = datetime.now()
benchmark_with_FPDF(tests, out_folder, in_folder)
total_time = datetime.now() - start_time
print(f"Benchmark took {str(total_time)}")
```
This gave me a runtime of `45.574701 seconds`. Note that the input was 24 PNG
images, generating a 24 page PDF.

The results from the intial profiling are captured below.
```python
Benchmark took 0:01:48.941374
Wrote profile results to benchmark.py.lprof
Timer unit: 1e-06 s

Total time: 108.941 s
File: .\benchmark.py
Function: benchmark_with_FPDF at line 20

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
    20                                           @profile
    21                                           def benchmark_with_FPDF(tests, out_folder, in_folder):
    22         1         59.7     59.7      0.0      pdf = FPDF(orientation='L', unit='mm', format='A4')
    23         3          1.6      0.5      0.0      for test in tests:
    24        24         25.5      1.1      0.0          for signal in SIGNALS_TO_CAPTURE:
    25        24         39.2      1.6      0.0              image = in_folder + "\\" + test + "\\"+ test + "_" + signal + ".png"
    26                                                       # Only add the graph to PDF if it exists.
    27        24       2324.0     96.8      0.0              if (os.path.isfile(image)):
    28        24        708.6     29.5      0.0                  pdf.add_page()
    29        24  108695906.0 4528996.1     99.8                  pdf.image(image, x=0, y=20, w=300, h=150)
    30                                                       else:
    31                                                           print("File: " + image + " does not exist.")
    32
    33
    34         1     242092.4 242092.4      0.2      pdf.output(out_folder + "\\FPDF_benchmark.pdf", "F")

```

According to the results, 99.8% of execution time is spent on this FPDF API
call.
```py
pdf.image(image, x=0, y=20, w=300, h=150)
```
My intuition seems to be correct; it's clear that the bottleneck is in FPDF.

## Alternatives to FPDF
After some research, I found the following tools to be candidates for replacing
FPDF.

- [`img2pdf`][4]
- [`Pillow`][5]

It seems that `img2pdf` actually uses `Pillow` internally. These `img2pdf` API
seems straightforward to use, so I'll just be testing that. 

The benchmarking function rewritten to use `img2pdf` is included below.
```python
@profile
def benchmark_with_img2pdf(tests, out_folder, in_folder):
    # Get a list of paths to the images
    paths = []
    for test in tests:
        for signal in SIGNALS_TO_CAPTURE:
            image = in_folder + "\\" + test + "\\"+ test + "_" + signal + ".png"
            # Only add the graph to PDF if it exists.
            if (os.path.isfile(image)):
                paths.append(image)
            else:
                print("File: " + image + " does not exist.")

    # Specify paper size (A4 landscape)
    a4inpt = (img2pdf.mm_to_pt(297), img2pdf.mm_to_pt(210))
    layout_fun = img2pdf.get_layout_fun(a4inpt)

    # Generate PDF
    with open(f"{out_folder}\\benchmark_img2pdf.pdf","wb") as f:
        f.write(img2pdf.convert(paths, layout_fun=layout_fun))
```

As before, we run the script again with `line_profile`.

### Results
Using my crude test script again, the img2pdf API only took 1.887416 seconds.
The `line_profiler` results confirm this:
```python
Total time: 94.2377 s
File: .\benchmark.py
Function: benchmark_with_FPDF at line 21

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
    21                                           @profile
    22                                           def benchmark_with_FPDF(tests, out_folder, in_folder):
    23         1        110.0    110.0      0.0      pdf = FPDF(orientation='L', unit='mm', format='A4')
    24         3          2.5      0.8      0.0      for test in tests:
    25        24         44.1      1.8      0.0          for signal in SIGNALS_TO_CAPTURE:
    26        24         63.1      2.6      0.0              image = in_folder + "\\" + test + "\\"+ test + "_" + signal + ".png"
    27                                                       # Only add the graph to PDF if it exists.
    28        24       3338.2    139.1      0.0              if (os.path.isfile(image)):
    29        24       1062.9     44.3      0.0                  pdf.add_page()
    30        24   93822206.6 3909258.6     99.6                  pdf.image(image, x=0, y=20, w=300, h=150)
    31                                                       else:
    32                                                           print("File: " + image + " does not exist.")
    33
    34
    35         1     410847.1 410847.1      0.4      pdf.output(out_folder + "\\FPDF_benchmark.pdf", "F")

Total time: 1.88704 s
File: .\benchmark.py
Function: benchmark_with_img2pdf at line 37

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
    37                                           @profile
    38                                           def benchmark_with_img2pdf(tests, out_folder, in_folder):
    39                                               # Get a list of paths to the images
    40         1          0.5      0.5      0.0      paths = []
    41         3          1.2      0.4      0.0      for test in tests:
    42        24          8.7      0.4      0.0          for signal in SIGNALS_TO_CAPTURE:
    43        24         25.2      1.1      0.0              image = in_folder + "\\" + test + "\\"+ test + "_" + signal + ".png"
    44                                                       # Only add the graph to PDF if it exists.
    45        24        938.6     39.1      0.0              if (os.path.isfile(image)):
    46        24         15.7      0.7      0.0                  paths.append(image)
    47                                                       else:
    48                                                           print("File: " + image + " does not exist.")
    49
    50                                               # specify paper size (A4 landscape)
    51         1          3.9      3.9      0.0      a4inpt = (img2pdf.mm_to_pt(297), img2pdf.mm_to_pt(210))
    52         1          9.8      9.8      0.0      layout_fun = img2pdf.get_layout_fun(a4inpt)
    53
    54                                               # Generate PDF
    55         1        408.9    408.9      0.0      with open(f"{out_folder}\\benchmark_img2pdf.pdf","wb") as f:
    56         1    1885623.0 1885623.0     99.9          f.write(img2pdf.convert(paths, layout_fun=layout_fun))
```

- FPDF took approximately 94.2377 seconds to complete the conversion process.
- img2pdf completed the same task in approximately 1.88704 seconds.

Therefore, by migrating our image to PDF conversion scripts to use img2pdf, we
should observe a 50x increase in PDF generations speeds. This is huge for
multi-hour batched simulations runs, which will have 50+ test cases across
multiple scenarios.

## Conclusion
There was a major bottleneck in our image to PDF conversion code. By moving
from `FPDF` to `img2pdf`, we saw a 50x increase in program execution speed.


[1]:https://docs.python.org/3/library/timeit.html
[2]:https://github.com/pyutils/line_profiler
[3]:https://github.com/psf/pyperf
[4]:https://github.com/myollie/img2pdf
[5]:https://pillow.readthedocs.io/en/stable/