transparently migrate data from local file storage to key-value-stores like RIAK KV

Posted on by Daniel Kirstenpfad

About 2 years ago I sat down and wrote a filesystem. Well not from scratch but using the great FUSE (Filesystem in Userspace) framework. I’ve released it as open-source later on Github:

This script acts as glue between a local file storage mount point and RIAK. It is targeted at specific use cases when local mount-points need to be migrated to RIAK without changing the applications accessing those mount point. Think of it as a transparent RIAK filesystem layer with multiple options to control it’s behavior regarding local files.

riakfuse

I had a very specific purpose in mind when I wrote this: There was a local filesystem that got filled up and because of technical restrictions we were not able to resize it or even completely copy it without interrupting the service using the data stored there.

Since we were already using the RIAK Key-Value Database for certain binary loads the idea came up to also utilize this key-value concept for a filesystem.

The idea now is: You have a local filesystem that holds a lot of folders and files already and you want to gradually want to move it to new grounds.

This migration needed to happen with minimal service interruptions assuming that there is constant reading and writing happening.

In this riak-fuse project I’ve written an overlay filesystem that steps between the application and the underlying “old”-filesystem. It looks and behaves identical to the application reading and writing.

But, depending on the mode you have chosen while mounting, every file read will at first be read from the “old”-filesystem and after successfull read stored into the key-value store.

On the next read it will be read from the key-value store directly.

The same applies for writing. Riak-fuse will write either to both, local storage and key-value store or just to the key-value store.

So in a nutshell: Data is slowly but surely on each access transferred over to the key-value store and load + storage space is slowly moving over from the local storage to the key-value store.

To facilitate this there are a lot of options for this script:

This all comes with ready-to-use docker and docker-compose files for you to try out.

Also it might interest you as an extremely simplified example of how to write an actual file system module for FUSE in Python.

Disclaimer: This effectively is my first python script as well as fuse module. Don’t be too hard on your judgement.

Things you do with Microsoft Excel

Posted on by Daniel Kirstenpfad

Many of us are happy when they can accomplish the most simple tasks with Excel without pulling their own hair out.

And then there are these people who do something entirely different with Excel:

Excel Unusual

Finding engineering work quite unchallenging lately I decided to start this blog in which to share cool ways of solving engineering problems or just interesting modeling of natural phenomena in MS Excel 2003. I use mainly cell formulas with minimum of VBA in order to take advantage of the ease of “programming” and the native speed of the Excel spreadsheet.

http://www.excelunusual.com/

Time estimation in software development

Posted on by Daniel Kirstenpfad

I’ve found myself in these spots several times in my life. Either I had to deliver on an estimate or I had to acknowledge an estimation and deal with the outcomes.

If you are involved in anything digital / software this is a recommended piece to read:

Anyone who built software for a while knows that estimating how long something is going to take is hard. It’s hard to come up with an unbiased estimate of how long something will take, when fundamentally the work in itself is about solving something. One pet theory I’ve had for a really long time, is that some of this is really just a statistical artifact.

Why software projects take longer than you think

“Around Tokyo” project

Posted on by Daniel Kirstenpfad

If you are interested in even some older video recordings, pictures and impressions about some areas of Tokyo this is a good time-sink for you.

Lyle Saxon has several old-fashioned looking websites (wonderful) with lots and lots of browsing content as well as a YouTube channel with recordings from earlier times:

Tokyo resident since 1984. Video material from 1990-93, as well as newer material from 2008 onwards.

The “Around Tokyo” project was and is to document life in Tokyo and the surrounding areas, as well as some material from other areas of Japan.

YouTube Channel

QR codes – how do they work?

Posted on by Daniel Kirstenpfad

I came across a very nice explanatory piece for QR codes. If you always wanted to know the basic principles this is a good chance to get a grasp.

QR code (abbreviated from Quick Response Code) is the trademark for a type of matrix barcode (or two-dimensional barcode) first designed in 1994 for the automotive industry in Japan. A barcode is a machine-readable optical label that contains information about the item to which it is attached. In practice, QR codes often contain data for a locator, identifier, or tracker that points to a website or application. A QR code uses four standardized encoding modes (numeric, alphanumeric, byte/binary, and kanji) to store data efficiently; extensions may also be used.[1]

Wikipedia
Robomatics QR code explanation

I am using QR codes in several of my projects – one example: Miataru uses QR codes to encode the device ID and help with the device handshake. You scan the QR code of your friend with your Miataru client app and immediately will be able to see his location in Miataru. Without the need to enter long rows of numbers.

convert Markdown scribbles to vector drawings

Posted on by Daniel Kirstenpfad

Every task you take
Every meeting you make
Every keyboard you break
Every note you take
I’ll be storing it for you

my text editor

Well that was fun! And indeed: a big portion of my professional daily business is taking place in a text editor taking notes and scribbling ideas and thoughts.

I’ve tried many things but the only way that resonated with me was taking notes in Markdown in a text editor that supports markdown. Currently that editor is Atom.io. Mainly because it is not in the way and quite portable. Runs on Linux, Windows, MacOS.

extra cheesy 80s neon theme

This way – I just took a count – I noted down 364.416 words in the last 1.5 years on my current job (equals to about 46 hours of average speed reading…).

Along side those simple text notes and bullet lists I am doing very simple tables as well as ASCII scribbles in Markdown as well. With the right tools it’s extremely easy and much faster than booting up the Powerpoint or worse.

When you have all in Markdown you then can freely stylesheet away and convert to handy PDF files as well. All even with embedded images if you so desire.

But even if you sit on that treasure trove of Markdown there comes the time when you wish you could convert your scribbles to graphics. Even if it is for the sole reason to not have to draw it again for that fancy Powerpoint slide deck.

You’ve got multiple options to accomplish this:

svgbob is at first a command line tool that got a recent level-up with a proper web-frontend:

When given Markdown it creates graphics. In the picture above the input is on the left and the svgbob output on the right (as SVG).

Markdeep is the alternative. Which of both work for your case depends on that specific case. Knowing and using both properly is the best way.

from now on console emulation will look much better

Posted on by Daniel Kirstenpfad

A modder going by the handle DerKoun has released an “HD Mode 7” patch for the accuracy-focused SNES emulator bsnes. In their own words, the patch “performs Mode 7 transformations… at up to 4 times the horizontal and vertical resolution” of the original hardware.

The results, as you can see in the above gallery and the below YouTube video, are practically miraculous. Pieces of Mode 7 maps that used to be boxy smears of color far in the distance are now sharp, straight lines with distinct borders and distinguishable features. It’s like looking at a brand-new game.

ArsTechnica

Michelin Guide Restaurants in Tokyo

Posted on by Daniel Kirstenpfad

You are likely aware of the existence of the “Michelin Guide“.

Michelin Guides are a series of guide books published by the French tyre company Michelin for more than a century. The term normally refers to the annually published Michelin Red Guide, the oldest European hotel and restaurant reference guide, which awards up to three Michelin stars for excellence to a select few establishments.

Wikipedia

You might also be aware that Tokyo is the city with the highest density of Michelin star rated restaurants. Nice, eh?

A purchase of this guide is recommended in any case but these days people also need something they can intuitively use and which integrates into already existing workflows.

These people, like myself, need a map and maybe more details in a machine readable, filterable spreadsheet.

And as time goes on it might be quite useful to have all the sources that lead to these great tables and maps. Sources that allow you to crawl and grab these information.

A script that crawls Tokyo-based michelin guide establishments and saves it into a JSON file. I personally did this project so I can plan my tokyo trip based on the cheapest and most-renowned restaurants,

Michelin Guide Crawler on GitHub

Swappiness is a thing, as is cache pressure

Posted on by Daniel Kirstenpfad

We know that using swap space instead of RAM (memory) can severely slow down the performance of Linux. So then, one might ask, since I have more than enough memory available, wouldn’t it better to remove swap space completely? The short answer is, No. There are performance benefits when swap is enabled, even when you have more than enough ram.

Why you should almost always add swap space

vfs_cache_pressure – Controls the tendency of the kernel to reclaim the memory which is used for caching of directory and inode objects. (default = 100, recommend value 50 to 200)

swappiness – This control is used to define how aggressive the kernel will swap memory pages. Higher values will increase aggressiveness, lower values decrease the amount of swap. (default = 60, recommended values between 1 and 60) Remove your swap for 0 value, but usually not recommended in most cases.

https://access.redhat.com/solutions/103833

As I’ve now brought up the topic, go ahead and read the complete story at the source.

Pointing and Calling (指差喚呼)

Posted on by Daniel Kirstenpfad

When you are travelling Japan you will observe very interesting things while using public transport. In a train or a bus the driver is likely to talk and seemingly magically point with his finger and wave his hand.

You will very likely observe a behavior that might not make sense at first but is fascinating to see. And all is to ensure the safety of the vehicle and all it’s passengers.

It might look like this:

CC-BY: Alan Levine from Strawberry, United States 

Pointing and calling is a method in occupational safety for avoiding mistakes by pointing at important indicators and calling out the status. It is common in Japan and railways of China. It is sometimes referred to by its Japanese terms, shisa kanko (指差喚呼), shisa kakunin kanko (指差確認喚呼) or yubisashi koshō (指差呼称).

Making large gestures and speaking out the status helps keeping focus and attention. The method was first used by train drivers and is now commonly used in Japanese industry.

It is recommended by the Japan Industrial Safety and Health Association (JISHA, 中央労働災害防止協会)

.
Pointing and calling requires co-action and co-reaction among the operator’s brain, eyes, hands, mouth, and ears.

Wikipedia

procedural generated traditional Chinese landscape scrolls

Posted on by Daniel Kirstenpfad

{Shan, Shui}* is inspired by traditional Chinese landscape scrolls (such as this and this) and uses noises and mathematical functions to model the mountains and trees from scratch. It is written entirely in javascript and outputs Scalable Vector Graphics (SVG) format.

https://github.com/LingDong-/shan-shui-inf

This is quite impressive and I am thinking about pushing that into the header of this blog :-) It’s just too nice looking to pass on.

file from the far future

Posted on by Daniel Kirstenpfad

I ran a VVV job to catalog a storage array I have. To my surprise at least one file had a very very strange timestamp:

Apparently the file in question was generated on an action cam which had lost its correct date and time setting at the time of recording…

The tool I am using to catalogue the storages is also worth a mention:

VVV is an application that catalogs the content of removable volumes like CD and DVD disks for off-line searching. Folders and files can also be arranged in a single, virtual file system. Each folder of this virtual file system can contain files from many disks so you can arrange your data in a simple and logical way.
 
VVV also stores metadata information from audio files: author, title, album and so on. Most audio formats are supported.

about VVV

let AI convert videos to comic strips for you

Posted on by Daniel Kirstenpfad

Artificial Intelligence is used more and more to achieve tasks only humans could do before. Especially in the areas that need a certain technique to be mastered AI goes above and beyond what humans would be able to do.

In this case a team has implemented something that takes video inputs and generates a comic strip from this input. Imagine it to look like this:

Input
Output

In this paper, we propose a solution to transform a video into a comics. We approach this task using a
neural style algorithm based on Generative Adversarial Networks (GANs).

Paper
click to read the paper

They even made a nice website you can try it yourself with any YouTube Video you want:

“Kowloon Walled City” themed arcade in Kawasaki (あなたのウエアハウス)

Posted on by Daniel Kirstenpfad

While we were visiting Japan we usually stay quite close to Kawasaki. And with some hints we found that a replication of “Kowloon Walled City” had been put up as a video game arcade there.

Kowloon Walled City was a largely ungoverned, densely populated settlement in Kowloon City, Hong Kong. Originally a Chinese military fort, the Walled City became an enclave after the New Territories was leased to Britain by China in 1898. Its population increased dramatically following the Japanese occupation of Hong Kong during World War II. By 1990, the walled city contained 50,000 residents within its 2.6-hectare (6.4-acre) borders. 

Wikipedia: Kowloon Walled City
OpenStreetMap

A partial recreation of the Kowloon Walled City exists in the Anata No Warehouse, an amusement arcade that opened in 2009 in the Japanese suburb of Kawasaki, Kanagawa. The designer’s desire to accurately replicate the atmosphere of the Walled City is reflected in the arcade’s narrow corridors, electrical wires, pipes, postboxes, sign boards, neon lights, frayed posters, and various other small touches that 

Wikipedia: Anata No Warehouse

I did not know a lot about the Kowloon Walled City before we found this arcade. And it’s – as you can imagine – a very colorful reproduction of the ambiance that you – according to documentations and reports from the time – would have experienced. Especially in the entrance area, the theming of the rooms and some game cabinets as well as for example the rest-rooms.

Of course there is a full blown quite nice but – as it is good custom – extremely noisy arcade in there as well. We’ve easily ‘lost’ 3 hours in there. Be aware that smoking is allowed in these places in Japan.

The first floor contained the UFO catcher machines and a good portion of vintage and modern arcade cabinets. I’ve had a go and Gradius and greatly enjoyed it. There’s a battery of Mech-Pods as well as racing and rythm games.

The second floor had lots of pachinko and other medal and slot machines. Even more noise than any arcade cabinets could do.

The third floor finally contains Dart and Snooker / Billard tables.

All in all it was one of the nicer arcades. Much nicer than others because there was a lot more room. It did not feel half als claustrophobic as an arcade usually feels in Japan.

Kawasaki Frontale vs. Vissel Kobe

Posted on by Daniel Kirstenpfad

First: I am not at all interested or knowledgeable in football / soccer. But…

Several times a year I am for multiple weeks in the area of Musashikosugi ((武蔵小杉) which is part of the greater Tokyo area in Japan.

And because of these stays – I’ve probably been there for the accumulated time of 1 year – I’ve got attached to this area / community over time.

This includes all those things the community shares on various places on the internet so to a small degree I can stay informed.

For example: There is a fantastic blog (as for many other communities in Tokyo) that specifically shares community related information about Musashikosugi.

To understand the context you need to know that I’ve worked for Rakuten. The number one eCommerce company in Japan. This surely kick-started my interest in Japan overall.

I know from my time at Rakuten that the company engaged in a couple of sport sponsorships. One of them was the J1 League football club “Vissel Kobe“.

Kawasaki, the area of which Musashikosugi is part of, also has such a football team called “Kawasaki Frontale“.

Through said blog it came to my attention that there was a game between those two football teams and…

Scoreboard at Ito-Yokado Musashi Kosugi Ekimae store (musuashikosugi blog)

Kawasaki won the game!

So despite me not being particularly interested in sports this news was quite exciting to see. It almost feels like some local patriotism feelings come up. And with the direct connection to my past employment it get’s even more exciting.

Go Kawasaki, go!