feat: new articles
This commit is contained in:
@@ -1,45 +0,0 @@
|
||||
---
|
||||
title: Extract Thumbnail Image from Affinity Photo and Affinity Design
|
||||
---
|
||||
|
||||
Nextcloud doesn't have a support for thumbnail generation from Affinity Photo and Affinity Design. So I had to do it myself.
|
||||
|
||||
# Digging Binary
|
||||
|
||||
Glancing at `.afphoto` and `.afdesign` in Finder, I noticed that it has a QuickLook support and an ability to show the thumbnail image. So these files should have thumbnail image somewhere inside its binary.
|
||||
|
||||
I wrote a simple script to seek for thumbnail image from a binary and save it as `.png` file.
|
||||
|
||||
```js af.js
|
||||
const fs = require("fs");
|
||||
|
||||
// png spec: https://www.w3.org/TR/PNG/
|
||||
const PNG_SIG = Buffer.from([137, 80, 78, 71, 13, 10, 26, 10]);
|
||||
const IEND_SIG = Buffer.from([73, 69, 78, 68]);
|
||||
|
||||
function extractThumbnail(buf) {
|
||||
const start = buf.indexOf(PNG_SIG);
|
||||
const end = buf.indexOf(IEND_SIG, start) + IEND_SIG.length * 2; // IEND + CRC
|
||||
return buf.subarray(start, end);
|
||||
}
|
||||
|
||||
function generateThumbnail(input, output) {
|
||||
const buf = fs.readFileSync(input);
|
||||
const thumbBuf = extractThumbnail(buf);
|
||||
fs.writeFileSync(output, thumbBuf);
|
||||
}
|
||||
|
||||
generateThumbnail(process.argv[2], process.argv[3] || "output.png");
|
||||
```
|
||||
|
||||
That's right. This script just scrapes a binary file and extracts the portion of which starts with `PNG` signature and ends with `IEND`.
|
||||
|
||||
Now I can generate a thumbnail image from arbitrary `.afphoto` and `.afdesign` file. Let's move on delving into Nextcloud source code.
|
||||
|
||||
# Tweaking Nextcloud
|
||||
|
||||
I have a little experience in tweaking Nextcloud source code before, where I implemented thumbnail generator for PDFs, so it should be easier this time, hopefully.
|
||||
|
||||
|
||||
|
||||
Anyway, long story short, I got Nextcloud generates thumbnail images for Affinity files by implementing PreviewGenerator class.
|
||||
Binary file not shown.
|
Before Width: | Height: | Size: 437 KiB |
@@ -1,36 +0,0 @@
|
||||
---
|
||||
title: Build Chromium from zero
|
||||
---
|
||||
|
||||
事前準備する。
|
||||
|
||||
```
|
||||
brew install cache
|
||||
git config --global core.precomposeUnicode true
|
||||
```
|
||||
|
||||
ソースコードを手に入れる。
|
||||
|
||||
```shell
|
||||
ghq get https://chromium.googlesource.com/chromium/tools/depot_tools.git
|
||||
cd `ghq root`/chromium.googlesource.com/chromium
|
||||
fetch chromium
|
||||
```
|
||||
|
||||
`.envrc` に以下を追加し、`direnv allow`で環境変数を適用する。
|
||||
|
||||
```shell
|
||||
PATH_add `qhq root`/chromium.googlesource.com/chromium/tools/depot_tools
|
||||
PATH_add src/third_party/llvm-build/Release+Asserts/bin
|
||||
export CCACHE_CPP2=yes
|
||||
export CCACHE_SLOPPINESS=time_macros
|
||||
export SPACESHIP_GIT_SHOW=false
|
||||
```
|
||||
|
||||
ビルドする。
|
||||
|
||||
```shell
|
||||
cd src
|
||||
gn gen out/Default --args='cc_wrapper="ccache"'
|
||||
autoninja -C out/Default chrome
|
||||
```
|
||||
@@ -1,23 +0,0 @@
|
||||
---
|
||||
title: 深圳を旅する Tips
|
||||
---
|
||||
|
||||
## WeChat Pay
|
||||
|
||||
深圳での食事は殆ど WeChat Pay で支払うことが出来た。
|
||||
|
||||
## UnionPay
|
||||
|
||||
ホテルやスターバックスで Visa カードを使うことが出来なかったため、UnionPay カードで支払うことになった。
|
||||
|
||||
## Pocketchange
|
||||
|
||||
日本の空港に設置している pocketchange で外貨や日本円を各種電子マネーに両替することが出来る。
|
||||
|
||||
## Google Translate
|
||||
|
||||
翻訳データはダウンロードしてローカルで使えるようにしておくこと。
|
||||
|
||||
## Octopus / 深圳通
|
||||
|
||||
KKday で事前予約していた物を香港空港で受け取った。深圳通は現地で購入した。
|
||||
@@ -1,12 +0,0 @@
|
||||
---
|
||||
title: Deconvolutionと呼ぶのはもうやめよう
|
||||
date: 2017-03-05 13:44:00 +09:00
|
||||
---
|
||||
|
||||
深層学習において、Convolutional Layer (畳み込み層)とは、あるシェイプのテンソルをそれ以下のサイズに縮約する性質のレイヤーです。一方で Deconvolution Layer (逆畳み込み層)とは、[Jonathan Long, et al](https://arxiv.org/abs/1411.4038)の論文で提案されたレイヤーで、あるシェイプのテンソルをそれ以上のサイズに拡大する性質を持ちます。
|
||||
|
||||
ところが実際のところ、このレイヤーは Transposed Convolution Layer (転置畳み込み層)と呼ぶべきです。なぜかを以下に示します。
|
||||
|
||||
> Upsampling is backwards strided convolution. (アップサンプリングは
|
||||
|
||||
[Stack Exchange](http://datascience.stackexchange.com/questions/6107/what-are-deconvolutional-layers)での議論を踏まえると
|
||||
@@ -1,30 +0,0 @@
|
||||
---
|
||||
title: Developing Web Apps in One Minutes
|
||||
---
|
||||
|
||||
## 0. Setup Homebrew and Node
|
||||
|
||||
```
|
||||
/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
|
||||
```
|
||||
|
||||
```
|
||||
brew install node
|
||||
```
|
||||
|
||||
## 1. Scaffold from template
|
||||
|
||||
```
|
||||
npx express-generator --view=pug awesome-app
|
||||
cd awesome-app
|
||||
npm install
|
||||
npm start
|
||||
```
|
||||
|
||||
## 2. Deploy with Now
|
||||
|
||||
```
|
||||
npm install -g now
|
||||
now login
|
||||
now --public
|
||||
```
|
||||
@@ -1,11 +1,11 @@
|
||||
---
|
||||
title: 'gst: a powerful pal for ghq'
|
||||
title: "gst: a powerful pal for ghq"
|
||||
date: 2017-06-02 23:02:00 +09:00
|
||||
---
|
||||
|
||||
[gst](https://github.com/uetchy/gst) is tiny and simple but powerful pal for [ghq](https://github.com/motemen/ghq).
|
||||
|
||||
Have you ever imagined what if you know which commits are unpushed or which changes are uncommitted yet, for all of the repositories you have cloned on your machine?
|
||||
Have you ever imagined what if you know which commits are unpushed or which changes are uncommitted yet, for all the repositories you have cloned on your machine?
|
||||
|
||||
You might want to check out my ongoing project `gst`:
|
||||
it might help you to know what ongoing changes are remained to be committed or pushed among entire your local repositories.
|
||||
|
||||
@@ -3,7 +3,7 @@ title: パケットキャプチャリング
|
||||
---
|
||||
|
||||
- macOS Mojave では、Wi-Fi をオフにしていないと Monitor モードでスキャンしてもパケットが受信できない。
|
||||
- Preferences > Protocols > IEEE 802.11 で Decrypt Keys を保存する。wpa-psk でハッシュ化された値を保存したほうが安全である。保存先は`.config/wireshark`
|
||||
- Preferences > Protocols > IEEE 802.11 で Decrypt Keys を保存する。しかし wpa-psk でハッシュ化された値を保存したほうが安全だ。保存先は`.config/wireshark`
|
||||
- 暗号化された 802.11 通信を覗くには 4-ways handshake (EAPOL)を観測する必要がある。そのためには対象デバイスの Wi-Fi をトグルすれば良い。
|
||||
|
||||
## コマンド
|
||||
|
||||
@@ -1,21 +0,0 @@
|
||||
---
|
||||
title: "[].map(parseInt)"
|
||||
---
|
||||
|
||||
## Fan fact
|
||||
|
||||
`[0xa, 0xa, 0xa].map(parseInt)` results in `[10, NaN, 2]`.
|
||||
|
||||
## Why???
|
||||
|
||||
`parseInt(0xa, 0, [0xa, 0xa, 0xa])`
|
||||
|
||||
The second argument is `0` so the first argument gonna be treated as decimal number becoming `10`.
|
||||
|
||||
`parseInt(0xa, 1, [0xa, 0xa, 0xa])`
|
||||
|
||||
The second argument is `1` which is invalid as a radix so the result ends up with `NaN`.
|
||||
|
||||
`parseInt(0xa, 2, [0xa, 0xa, 0xa])`
|
||||
|
||||
The second argument is `2` meaning the first argument going to be handled as a binary number. `0xa` is `10` in binary, which results in `2` in decimal form.
|
||||
@@ -1,14 +0,0 @@
|
||||
---
|
||||
date: 2020-02-13 16:22:05 +0900
|
||||
title: 静寂を得る方法
|
||||
---
|
||||
|
||||
聴覚過敏であったり、そうでなくとも周りの音がパフォーマンスに悪影響を与える人のために、静寂を得る方法を紹介します。
|
||||
|
||||
## EARIN M-2
|
||||
|
||||
[EARIN](https://earin.com/) は左右分離型 Bluetooth イヤホンです。付属のイヤホンの代わりに自分の耳にフィットする Comply のイヤーチップと付け替えます。
|
||||
|
||||
## Moldex
|
||||
|
||||
Moldex は使い捨て耳栓のメーカーであり、各種遮音レベルに分かれた多様なラインナップを提供しています。
|
||||
@@ -1,21 +1,21 @@
|
||||
---
|
||||
title: Toxicity Analysis in Vtuber Live Chat
|
||||
title: Toxicity Analysis in YouTube Live Chat
|
||||
---
|
||||
|
||||
A little exploration and experiment on classifying toxic comments.
|
||||
A little exploration and experiment on toxic activities.
|
||||
|
||||
# Why
|
||||
|
||||
The motivation is simple; I just feel sad when they look suffered from toxic comments in live chats. The goal is also simple: design an automated system to spot toxic comments and destroy them.
|
||||
The motivation is quite simple; I just feel sad when they sound suffered from toxic chats. The goal is also simple: design an automated system to spot toxic chat and quarantine them.
|
||||
|
||||
# Data Data Data
|
||||
# Data, Data, Data
|
||||
|
||||
> I can't make bricks without clay.
|
||||
> I can't make bricks without clay.
|
||||
> — Sherlock Holmes
|
||||
|
||||
I need a myriad of live chat comments and moderation events for analysis and future use.
|
||||
|
||||
Unfortunately, YouTube API does not offer a way to retrieve these kind of events in real time. Which is so crucial because live streams are only place we can observe moderators' activities through API response. Once it gets archived, these events are no longer available.
|
||||
Unfortunately, YouTube API does not offer a way to retrieve these kinds of events in real time. Which is so crucial because live streams are only place we can observe moderators' activities through API response. Once it gets archived, these events are no longer available.
|
||||
|
||||
## Collecting Crusts
|
||||
|
||||
@@ -29,17 +29,17 @@ collector <videoId>
|
||||
|
||||
A line with white text is a normal chat, with red text is a ban event, with yellow text is a deletion event.
|
||||
|
||||
## Make the Bread Rise
|
||||
## Make a Bread Rise
|
||||
|
||||
I know, that's not scalable at all. A new live stream comes in, I copy and paste video id into the terminal and run the script. How sophisticated.
|
||||
|
||||
Thankfully, there's a fantastic service around Hololive community: [Holotools](https://hololive.jetri.co). They operates an API that gives us past, ongoing, and upcoming live streams from Hololive talents.
|
||||
Thankfully, there's a great web service around Hololive community: [Holotools](https://hololive.jetri.co). They operate an API that gives us an index of past, ongoing, and upcoming live streams from Hololive talents.
|
||||
|
||||
Here I divided my system into two components: watch tower and collection worker. Watch tower periodically checks for newly scheduled live streams through Holotools API and create a job to be handled by workers. Collection workers are responsible for handling jobs and spawning a process to collect live chat events.
|
||||
Here I divided my system into two components: Scheduler and workers. Scheduler periodically checks for newly scheduled live streams through Holotools API and create a job to be handled by workers. Workers are responsible for handling jobs and spawning a process to collect live chat events.
|
||||
|
||||
|
||||
|
||||
I run the cluster for a while and by far it collects approximately 1 million comments per day. Now I could reliably run my own bakery.
|
||||
I run the cluster for a while and by far it hoards approximately 1 million comments per day. Now I could reliably run my own bakery.
|
||||
|
||||
# Look Before You Leap
|
||||
|
||||
@@ -51,23 +51,23 @@ Okay take a close look at the data before actually starting to build a model.
|
||||
|
||||
## By language
|
||||
|
||||
# Making Dataset
|
||||
# Creating Dataset
|
||||
|
||||
## Labelling Spam & Toxic Chat
|
||||
|
||||
### Utilizing Moderators' Activities
|
||||
|
||||
### Introducing Balanced Collocation Entropy
|
||||
### Introducing Normalized Co-occurrence Entropy
|
||||
|
||||
$$
|
||||
BCE(T) = \frac{N_T}{RLE_{string}(BWT(T))}
|
||||
NCE(T) = \frac{N_T}{RLE_{string}(BWT(T))}
|
||||
$$
|
||||
|
||||
$$
|
||||
BWT[T,i] = \begin{cases} T[SA[i]-1], & \text{if }SA[i] > 0\\ \$, & \text{otherwise}\end{cases}
|
||||
$$
|
||||
|
||||
Shannon Entropy is not enough. So I decided to combine the ideas of [Burrows-Wheeler Transform](https://en.wikipedia.org/wiki/Burrows%E2%80%93Wheeler_transform) and [Run-length Encoding](https://en.wikipedia.org/wiki/Run-length_encoding) and create a new entropy which represents "spamness" better than Shannon entropy does.
|
||||
Shannon Entropy is not enough. So I combined the ideas of [Burrows-Wheeler Transform](https://en.wikipedia.org/wiki/Burrows%E2%80%93Wheeler_transform) and [Run-length Encoding](https://en.wikipedia.org/wiki/Run-length_encoding) to formulate a new entropy which represents "spamminess" of given text.
|
||||
|
||||
### Browser Extension
|
||||
|
||||
@@ -85,17 +85,4 @@ Here's a [t-SNE](https://en.wikipedia.org/wiki/T-distributed_stochastic_neighbor
|
||||
|
||||
# Future
|
||||
|
||||
# Omake
|
||||
|
||||
## Hololive Dataset
|
||||
|
||||
I made collected chat events publicly available for those interested in further research.
|
||||
|
||||
The dataset contains:
|
||||
|
||||
- Chats
|
||||
- Superchats (amount, currency)
|
||||
- Retraction events
|
||||
- Moderation events (ban, delete)
|
||||
|
||||
## Toxicity Estimator Pre-trained Model
|
||||
When it's ready, I'm going to publish a dataset and pre-trained model used in this experiment.
|
||||
|
||||
@@ -1,40 +0,0 @@
|
||||
---
|
||||
title: 新しい自鯖
|
||||
---
|
||||
|
||||
10年ぶりに新しいサーバーを調達しました。最後に自鯖を組んだ時は高校生、沖縄に住んでいた頃です。BTOのタワーPCにDebianを入れてWebサーバーにしていました。UPSとか無いので台風で停電するたびにWebサービスが落ちるヘボ感です。
|
||||
|
||||
今回も完成品を買ってしまえばそれでお終いですが折角ですし、なにより面白そうなのでパーツからサーバーを組み立てましょう。初めてのAMD、初めてのDDR4メモリ、初めてのNVM Expressです。
|
||||
|
||||
# スペック
|
||||
|
||||
用途を考えましょう。
|
||||
|
||||
- 機械学習サーバー
|
||||
- 自宅クラウド
|
||||
- メールサーバー
|
||||
- ファイルサーバー (Nextcloud)
|
||||
- VPNサーバー他
|
||||
- VS Code Remote SSHのホスト先
|
||||
- 重いmakeやらなんやら
|
||||
- TabNine
|
||||
- Webサーバー
|
||||
- WebアプリやTelegram botのデプロイ先
|
||||
|
||||
重いタスクを並列してやらせたいので最優先はCPUとメモリです。メモリはデュアルリンクを重視して32GBx2を、CPUは昨今のライブラリのマルチコア対応を勘案してRyzen 9 3950Xにしました。
|
||||
|
||||
> 結果から言うとメモリはもっと必要でした。巨大なPandasデータフレームを並列処理なんかするとサクッと消えてしまいます。予算に余裕があるなら128GBほど用意したほうが良いです。
|
||||
|
||||
GPUは古いサーバーに突っ込んでいたNVIDIA GeForce GTX TITAN X (Maxwell)を流用しました。メモリが12GBありますが、最大ワークロード時でも5GBは残るので今のところ十分です。
|
||||
|
||||
記憶装置は3TB HDD 2台と500GB NVMeメモリです。NVMeメモリはOS用、HDDはデータとバックアップ用です。
|
||||
|
||||
マザーボードはASRockのB550 Taichiです。X570マザーと比較して、実装されているコンデンサーや安定性でB550にしました。
|
||||
|
||||
今後GPUを追加することを考えて800W電源を選びました。実際にサーバーを稼働させて使用電力を計測してみると、アイドル時に180W前後、フル稼働時でも350Wを超えないくらいでした。今後UPSを買う場合はその付近+バッファのグレードを買うと良いかもしれません。
|
||||
|
||||
ケースはFractal DesignのMeshify 2にしました。シンプルで良い。
|
||||
|
||||
OSは長年親しんできたDebian系を卒業してArchlinuxにしてみました。すでにファンになりかけています。本当に何も用意してくれません。セットアップウィザードとかないです。いきなりシングルユーザーモードにぶち込まれます。`which`すらインストールしなければ使えません。潔癖症の自覚がある人はArchを使いましょう。あとAURにいくつかパッケージを公開してみたので、よければvoteお願いします。
|
||||
|
||||
Arch Linuxのセットアップは個別に記事を書いたので読んでください。入力したコマンドを全て記録したので再現性があります。
|
||||
Reference in New Issue
Block a user