Daniel Flannery

Scraping PDFs from Flipsnack

Mon, 19 Sep 2022 00:00:00 +0000

Flipsnack is a service which allows you to upload PDFs and serve them as a flip book widget.

Create, share and embed online page flip catalogs, transforming your PDFs into online flipping books. Make a flip book online using our advanced flip book maker.

This is all well and good, but what if you want to download the content? Here is a hacky way to do that.

I run all of these steps within an Ubuntu WSL environment. What we are going to do:

Identify the CDN URL.
Download each of the pages as JPG images.
Rename the downloaded files to ensure the PDF output page order is correct.
Convert to PDF.
Bonus: Make the text selectable within the PDF.

The first thing you’ll want to do is grab the URL for the PDF you are looking to download. You can grab this using Dev tools in your browser and having a look at the Sources. The URL will have a structure containing a UNIQUE_ID and HASH_VALUE. Also notice the PAGE_NUMBER. Take note of the last available page as it’ll be needed when downloading.

https://cdn.flipsnack.com/collections/items/<UNIQUE_ID>/covers/page_<PAGE_NUMBER>/original?v=<HASH_VALUE>

Now let’s install the tools we need.

# Install the needed components
# Assuming Python 3+ is already installed
sudo apt install curl rename tesseract-ocr -y
pip install ocrmypdf

Next, we are going to use curl to download each of the images (pages) as JPGs. Here is where you want to pass in the page range. In this case, the PDF had 162 pages.

# Grab the images
curl https://cdn.flipsnack.com/collections/items/<UNIQUE_ID>/covers/page_\[1-162\]/original\?v\=<HASH_VALUE> -o "#1.jpg"

Once finished, you should now have a directory full of images, each of which is a page of the PDF.

├── 10.jpg
├── 11.jpg
├── 12.jpg
├── 13.jpg
├── 14.jpg
├── 15.jpg
├── 16.jpg
├── 17.jpg
├── 18.jpg
├── 19.jpg
├── 1.jpg
├── 20.jpg
├── 21.jpg
...

Since the order of images is incorrect, let’s add some leading zeros to the filenames using rename.

# Add 3 leading zeros to the filenames
rename 's/\d+/sprintf("%03d",$&)/e' *.jpg

That’s better.

├── 001.jpg
├── 002.jpg
├── 003.jpg
├── 004.jpg
├── 005.jpg
├── 006.jpg
├── 007.jpg
├── 008.jpg
├── 009.jpg
├── 010.jpg
├── 011.jpg
├── 012.jpg
├── 013.jpg
├── 014.jpg
├── 015.jpg
├── 016.jpg
├── 017.jpg
├── 018.jpg
├── 019.jpg
├── 020.jpg
├── 021.jpg
...

Time to merge these images into a PDF using the Python module img2pdf

# Merge and convert to PDF
python3 -m img2pdf *.jp* --output combined.pdf

As a final step, we can use the ocrmypdf module to add an OCR text layer to the PDF. This allows text to be searched or copy-pasted. This can take some time depending on the size of the PDF, so let it do its thing.

# Add OCR layer to PDF
python3 -m ocrmypdf combined.pdf combined_ocr.pdf

And with that, you have an offline copy of the PDF. There might be a much easier way of going about this, but it was a good excuse to play around with some tools.

How I make coffee

Sun, 05 Jun 2022 00:00:00 +0000

Coffee. Sweet black nectar of the gods. Here’s how I make it.

Equipment

An Aeropress
Aeropress paper filters
Hario Slim grinder
Digital scales
Spray bottle for water (Not critical)
Some good locally roasted coffee (Some I like)

Workflow

Aeropress is set up as standard. Over cup, on a scales with a paper filter.

Set Hario Slim grinder @ 8 clicks. For grind size, see below
Measure 13g coffee. Spray beans with water before grind to avoid static
Grind beans and add to Aeropress
Add boiling water to ~215g weight
Seal the Aeropress with plunger (total weight is now ~300g)
Let sit for 2 minutes
Swirl. Let sit for a further 30 seconds
Plunge with natural pressure. (~30 seconds)
Top-up cup with water if needed. (Total coffee weight usually ends up around 300g)
Enjoy ☕

Grind sizes for Hario Slim

The Hario Slim is a great, affordable hand grinder that I’ve been using almost daily without issue. For an idea on how the grinder settings map to what you are brewing, I use this table:

Type	Number of Clicks
Standard Drip Brew	10 clicks
#2 Pour over	10 clicks
Aeropress	6-8 clicks (10 for inverted possibly)
Espresso	5 clicks
French Press	12-14 clicks
Moka pot	9 clicks
Chemex	9 clicks

This is taken from a good guide here.

The way to measure your grind is by “clicks” away from all the way tight. You’ll feel the nut click as you unscrew, so count the number of clicks. The number of clicks to unscrew should vary based on the brew method you’re using.

James Hoffmann has a great walkthrough of the different methods on his YouTube channel. I would recommend checking it out. What I do here is essentially follow his guide. With this, I can make consistently good coffee that is on par or better than the local coffee shops, all while being much cheaper 👍.

Simulate CPU load with Python

Sat, 15 Jan 2022 00:00:00 +0000

While testing out some home automation code on my Raspberry Pi, I noticed it was pretty CPU intensive. Time to bump up the overclock to squeeze more performance out of the Broadcom Arm7 processor. I wanted to keep an eye on temperatures, as well as stability under full load, so I needed to simulate CPU usage.

This Python script will do the job. It uses the multiprocessing library, which you can read more about here.

#!/usr/bin/env python
from multiprocessing import Pool
from multiprocessing import cpu_count
import signal

stop_loop = 0

def exit_chld(x, y):
    global stop_loop
    stop_loop = 1

def f(x):
    global stop_loop
    while not stop_loop:
        x*x

signal.signal(signal.SIGINT, exit_chld)

if __name__ == '__main__':
    processes = cpu_count()
    print('-' * 20)
    print('Running load on CPU(s)')
    print('Utilizing %d cores' % processes)
    print('-' * 20)
    pool = Pool(processes)
    pool.map(f, range(processes))

This will utilize each CPU core and produce ~100% load with the while loops calculation.

--------------------
Running load on CPU(s)
Utilizing 8 cores
--------------------

Checking fail2ban jails

Sun, 14 Nov 2021 00:00:00 +0000

Fail2ban is an excellent tool to help ban unwelcome IPs from hitting your services.

Fail2ban scans log files (e.g. /var/log/apache/error_log) and bans IPs that show the malicious signs – too many password failures, seeking for exploits, etc. Generally Fail2Ban is then used to update firewall rules to reject the IP addresses for a specified amount of time, although any arbitrary other action (e.g. sending an email) could also be configured. Out of the box Fail2Ban comes with filters for various services (apache, courier, ssh, etc).

This bash script will print the current status of any fail2ban jails on the system.

#!/bin/bash
# Check the status of the Fail2Ban jails
JAILS=$(fail2ban-client status | grep "Jail list" | sed -E 's/^[^:]+:[ \t]+//' | sed 's/,//g')
for JAIL in $JAILS
do
    fail2ban-client status "$JAIL"
done

Output:

root@pihole:~ λ ./fail2banstatus.sh

Status for the jail: sshd
|- Filter
|  |- Currently failed: 2
|  |- Total failed:     18
|  `- File list:        /var/log/auth.log
`- Actions
   |- Currently banned: 2
   |- Total banned:     4
   `- Banned IP list:   <redacted IPs...>

This particular output is from a Pihole. For more information on configuring jails for a Pihole instance, see this forum post over on the Pihole discourse.

I tend to run this script using Ansible across all my local machines that have external services exposed. I’m hoping to detail that in an upcoming blog post.

My other website

Fri, 03 Sep 2021 00:00:00 +0000

My other website