Tebako: An executable packager (for Ruby programs)

Published at

5 days ago

Main Article

= Tebako: An advanced image packager for interpretive languages

Platform tests on GitHub: image:https://github.com/tamatebako/tebako/actions/workflows/ubuntu.yml/badge.svg["Ubuntu amd64", link="https://github.com/tamatebako/tebako/actions/workflows/ubuntu.yml"] image:https://github.com/tamatebako/tebako/actions/workflows/alpine.yml/badge.svg["Alpine", link="https://github.com/tamatebako/tebako/actions/workflows/alpine.yml"] image:https://github.com/tamatebako/tebako/actions/workflows/macos.yml/badge.svg["macOS amd64", link="https://github.com/tamatebako/tebako/actions/workflows/macos.yml"] image:https://github.com/tamatebako/tebako/actions/workflows/windows-msys.yml/badge.svg["Windows msys", link="https://github.com/tamatebako/tebako/actions/workflows/windows-msys.yml"]

Platform tests on Cirrus: image:https://api.cirrus-ci.com/github/tamatebako/tebako.svg?branch=main&task=ubuntu-aarch64["Ubuntu aarch64", link="https://cirrus-ci.com/github/tamatebako/tebako"]

Tools tests on GitHub: image:https://github.com/tamatebako/tebako-ci-containers/actions/workflows/build-containers.yml/badge.svg["Tebako cobtainers", link="https://github.com/tamatebako/tebako-ci-containers/actions/workflows/build-containers.yml"]

Quality: image:https://github.com/tamatebako/tebako/actions/workflows/lint-and-rspec.yml/badge.svg["lint and rspec", link="https://github.com/tamatebako/tebako/actions/workflows/lint-and-rspec.yml"] image:https://codecov.io/gh/tamatebako/tebako/graph/badge.svg?token=XD3emQ5qsY["Tebako cli rspec coverage", link="https://codecov.io/gh/tamatebako/tebako"]

== Purpose

Tebako is an advanced executable packager designed for applications written in interpretive languages.

It simplifies distribution and deployment by packaging your entire project with a bundled runtime into a single, performant, executable binary.

== Architecture

A Tebako package is effectively a self-executing container-in-a-file.

The package contains the following components:

An on-file filesystem (OFFS) containing all the project files and dependencies in DwarFS format ("application")
A runtime environment that includes the the necessary libraries and interpreters, with patched filesystem calls that redirect access of project files to the on-file filesystem ("runtime")

Tebako is capable to create a single file that contains both runtime and application or place runtime and application to separate files. In the latter case the runtime can be used with different applications or versions of the same application.

Please refer to mode option below that controls Tebako output.

== Supported runtimes, platforms and architectures

Tebako artifacts can be built and executed on the following platforms and architectures.

.Supported platforms and architectures [cols="3", options="header"] |=== | Platform and version | Architectures | Build system

|===

[NOTE]

Windows build caveats:

Tebako may face errors related to CMake path length limitations (https://gitlab.kitware.com/cmake/cmake/-/issues/25936). This error may affect not Tebako itself but the gems that need to be package and use CMake to build native extensions. There is no workaround for this issue as it looks like is a limitation of the manifest used to build CMake executable.
MSys strip utility creates broken executable when Tebako image is processed. Linking with -s flag produces unusable executables as well. Until this issue (https://github.com/tamatebako/tebako/issues/172) is resolved we plan to produce a Windows executable with debug information unstripped. You can opt to run strip -S manually, it most cases it works.

MacOS build caveats:

We saw clang compiler segmentation fault when packaging of very large projects with XCode 14.3.1. This issue is not reproducible with XCode 15.0.1 or higher.

.Supported Ruby versions [cols="2", options="header"] |=== | Ruby version | Supported platforms

| 2.7.8 | Linux, macOS | 3.0.7 | Linux, macOS | 3.1.6 | Linux, macOS, Windows | 3.2.{4,5,6,7} | Linux, macOS, Windows | 3.3.{3,4,5,6,7} | Linux, macOS, Windows | 3.4.1 | Linux, macOS, Windows

|===

NOTE: Our goal is to support all maintained Ruby releases, including minor versions.

== Package portability

=== General

Tebako packages are designed to be "forward portable" across different operating systems and architectures to allow for easy distribution and deployment.

Forward portability means that a package created on a specific platform can be executed on a newer version of the same platform.

=== macOS

macOS packages are forward portable across different macOS versions.

[example] A Tebako executable package built on macOS 13 (Ventura) can be executed on macOS 14 (Sonoma), but not vice versa.

x86_64 macOS packages can be run on Apple M (ARM) systems.

=== Linux distributions using musl

Packages built for the https://musl.libc.org[`musl` implementation of the C standard library] (such as https://alpinelinux.org[Alpine Linux]) are forward portable.

[example] A Tebako executable package built on Alpine 3.17 can be executed on Alpine 3.19.

Usage of the https://github.com/tamatebako/tebako-ci-containers[Tebako Docker containers] for packaging is encouraged since it eliminates the effort needed for toolchain setup and configuration.

=== Linux distributions using glibc

Packages built for the https://sourceware.org/glibc[`glibc` implementation of the C standard library] are forward portable if the --patchelf experimental option is enabled.

The --patchelf option allows these packages to be portable to Linux GNU distributions with GLIBC version 2.31 and above.

[example] A Tebako executable package built on Ubuntu 20.04 with --patchelf option can be executed on Rocky Linux 9.

Usage of the https://github.com/tamatebako/tebako-ci-containers[Tebako Docker containers] for packaging is encouraged since it eliminates the effort needed for toolchain setup and configuration.

.Minimum versions of GLIBC Linux distributions that support Tebako packages with forward portability [cols="3", options="header"] |=== | Distribution | Minimal supported version | GLIBC version

|===

== Future plans

Downloading new DwarFS images to be stored in the local home directory
Allowing loading multiple DwarFS images in a stacked way
Supporting a COW mechanism that the newly written files are stored in a separate image that can be loaded on top of the read-only file systems.

== FAQ

=== Why use Tebako?

Tebako is particularly useful for developers who need to:

Distribute applications without requiring users to have specific runtimes installed.
Simplify the deployment process by packaging all dependencies into one binary.
Ensure consistency across different environments by using a single executable.
Flexibility to support different runtime versions on the user's machine.

=== How do I know I need Tebako?

You might need Tebako if you:

Want to package your application into a single, self-contained binary.
Want to avoid the complexities of managing runtime environments on target machines.
Distribute software to environments where installing runtimes and their dependencies is challenging.
Require a streamlined way to deliver applications to end-users.
Need to ensure that your application runs consistently across different environments and architectures.

=== What is DwarFS?

https://github.com/mhx/dwarfs[DwarFS] is a fast, high compression read-only user-land file system designed for efficient storage and access of large collections of files.

It is used by Tebako to package applications into a compact and efficient format.

Tebako uses https://github.com/tamatebako/libdwarfs[libdwarfs], the library form of https://github.com/mhx/dwarfs[DwarFS], developed for the Tebako project.

=== When is Tebako better than comparable solutions?

Tebako offers several advantages over comparable solutions for supported interpretive languages.

They are listed in order of the degree of virtualization below.

Tebako stands out by providing a lightweight runtime bundling approach that simplifies distribution and deployment while offering flexibility and efficiency.

It eliminates the need for users to have specific runtimes installed and ensures consistency across different environments.

With Tebako, you can package your entire project with a bundled runtime into a single, performant, executable binary.

[cols="a,3a,3a"] |=== | Solution | Pros | Cons

| Virtual machines (VMs) |

Provides full isolation and compatibility across environments |
Requires a separate VM installation for each application
Heavy resource consumption for virtualization

| Docker |

Provides portable containers
Isolates entire applications and their dependencies
Supports easy deployment and scalability |
Requires Docker installation and management
Requires administrative rights on machine
Containerization overhead

| Tebako |

Packages all files and dependencies into a single binary
Supports multiple operating systems and architectures
Provides efficient packaging and execution with DwarFS
Offers security features like signing on macOS
Simplifies distribution and deployment
Native running speed |
Initial packaging time longer than Ruby gems
Minor runtime overhead

| Ruby Gems |

Easy installation of Ruby libraries
Provides user-side version control and dependency management |
Requires Ruby installation and gem management
Runtime execution dependent on the user's installed Ruby version and gems

|===

== Usage

=== Command-line interface

Tebako works by packaging your project into a single executable binary that includes all the necessary dependencies.

The way to work with Tebako is through its command-line interface (CLI). It provides the following commands:

setup:: Prepares the Tebako packaging environment.

press:: Packages a project into a single executable binary.

clean:: Removes Tebako artifacts.

clean_ruby:: Removes Tebako Ruby artifacts.

hash:: Calculates the Tebako script hash for use as a cache key in CI/CD environments.

version:: Displays the Tebako version.

help:: Displays the help message.

== Usage

=== General

Tebako can be used in two ways:

Through the Tebako container
Local installation

Please refer to the <> section on how to install Tebako.

[[installation]] == Installation

=== General

Installation of Tebako is only needed in order to package an application.

There is no need to install anything for users who run the packaged application.

=== Using Docker

==== General

If you have Docker installed and available, the easiest way to run Tebako is through the official Docker containers.

Docker containers with preinstalled Tebako packaging environments for Ubuntu and Alpine Linux are available at https://github.com/tamatebako/tebako-ci-containers[tebako-ci-containers].

==== Pull the container

Pull the Tebako container image.

[source,sh]

docker pull ghcr.io/tamatebako/tebako-:latest

<container_tag>:: is the desired image tag (e.g., ubuntu-20.04 or alpine-3.17).

==== Running Tebako commands in the container

Simply prefix the Tebako command with docker run and the container image.

[source,sh]

docker run -v :/mnt/w
-t ghcr.io/tamatebako/tebako-:latest
tebako {command} {parameters}

==== Packaging from outside the container

To package your application from outside the container, just run a single Docker command.

This command mounts the application folder into the container and runs the tebako press command, specifying the application root, entry point, output location, and Ruby version.

[source,sh]

docker run -v :/mnt/w
-t ghcr.io/tamatebako/tebako-:latest
tebako press

<application_folder>:: is the path to your application folder.

<container_tag>:: is the desired image tag (e.g., ubuntu-20.04 or alpine-3.17).

[example]

Assume that you have a Ruby application in the fontist folder of the current directory.

You can package it to ./fontist-package using the following command:

[source,sh]

docker run -v $PWD:/mnt/w
-t ghcr.io/tamatebako/tebako-ubuntu-20.04:latest
tebako press --root=/mnt/w/fontist --entry-point=fontist --output=/mnt/w/fontist-package --Ruby=3.2.4

====

==== Packaging from inside the container

It is also possible to package an application from inside the Tebako container.

Start and enter the container interactively.

[source,sh]

docker run -it --rm -v :/mnt/w
ghcr.io/tamatebako/tebako-:latest bash

<application_folder>:: is the path to your application folder.

<container_tag>:: is the desired image tag (e.g., ubuntu-20.04 or alpine-3.17).

Once inside, run the tebako press command:

[source,sh]

tebako press

[example]

Assume that you have a Ruby application in the fontist folder of the current directory.

You can package it to ./fontist-package using the following command:

[source,sh]

$ docker run -it --rm -v $PWD:/mnt/w ghcr.io/tamatebako/tebako-:latest bash

Inside the container:

$ tebako press --root=/mnt/w/fontist --entry-point=fontist --output=/mnt/w/fontist-package --Ruby=3.2.4

====

=== Local installation

==== General

There are cases where Docker may not be suitable for your needs, such as:

. Admin privileges: Running Docker requires administrative privileges, which means Docker may not be available to users on their machines.

. Performance penalty: Docker introduces a performance penalty due to the overhead of running containers. This can be a concern when packaging complex applications that require heavy memory usage.

In such cases, you can choose to install Tebako locally.

Tebako is distributed as a Ruby gem. A Ruby environment is necessary.

[source,sh]

$ gem install tebako

==== Prerequisites

These prerequisites are needed only for users who want to install Tebako on their machine and build all Tebako components locally.

If you use Docker, there is no need to set up these prerequisites.

===== Ubuntu 20.04

====== General

There are several prerequisites that need to be installed on Ubuntu 20.04 for Tebako to work correctly.

====== GNU C/C++ 10+ or Clang C/C++ 12+

[source,sh]

apt install -y gcc-10 g++-10 update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-10 10 update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-10 10

[source,sh]

apt install -y clang-12 update-alternatives --install /usr/bin/clang clang /usr/bin/clang-12 150 update-alternatives --install /usr/bin/clang++ clang++ /usr/bin/clang++-12 150

====== CMake version 3.20+

Tebako requires CMake at a version of at least 3.20+.

If such CMake version is not available as a default package, set it up as follows.

.Installing CMake 3.20+ [source,sh]

apt-get remove --purge --auto-remove cmake apt-get update apt-get install -y software-properties-common lsb-release curl apt-get clean all curl https://apt.kitware.com/kitware-archive.sh | bash apt-get install cmake

====== Other development tools and libraries

[source,sh]

apt-get -y install sudo git curl build-essential pkg-config bison flex autoconf
binutils-dev libevent-dev acl-dev libfmt-dev libjemalloc-dev libiberty-dev
libdouble-conversion-dev liblz4-dev liblzma-dev libssl-dev libunwind-dev
libboost-filesystem-dev libboost-program-options-dev libboost-system-dev
libboost-iostreams-dev libboost-date-time-dev libboost-context-dev
libboost-regex-dev libboost-thread-dev libbrotli-dev libdwarf-dev libelf-dev
libgoogle-glog-dev libffi-dev libgdbm-dev libyaml-dev libncurses-dev
libreadline-dev libncurses-dev libreadline-dev ruby-dev ruby-bundler
libutfcpp-dev

===== Alpine 3.17

====== General

There are several prerequisites that need to be installed on Alpine 3.17 for Tebako to work correctly.

====== APK packages

Run the following command to install all prerequisites.

[source,sh]

apk --no-cache --upgrade add build-base cmake git bash autoconf boost-static
boost-dev flex-dev bison make binutils-dev libevent-dev acl-dev sed python3
pkgconfig lz4-dev openssl-dev zlib-dev xz ninja zip unzip curl libdwarf-dev
libunwind-dev gflags-dev elfutils-dev libevent-static openssl-libs-static
lz4-static xz-dev zlib-static libunwind-static acl-static tar libffi-dev
gdbm-dev yaml-dev yaml-static ncurses-dev ncurses-static readline-dev
readline-static p7zip ruby-dev gcompat gettext-dev gperf brotli-dev
brotli-static jemalloc-dev fmt-dev xz-static

===== macOS

====== General

There are several prerequisites that need to be installed on macOS for Tebako to work correctly.

The following instructions work for:

macOS 13 (Ventura) through macOS 15 (Sequoia)

====== Homebrew packages

We use Homebrew to install the necessary packages on macOS.

Tebako provides a Brewfile at the repository root that you can install all dependencies using the brew bundle command.

[source,sh]

$ brew bundle

Alternatively, you can install the packages manually.

[source,sh]

$ brew update $ brew install gnu-sed bash pkg-config bison flex binutils libffi gdbm zlib
ncurses double-conversion boost jemalloc fmt glog libevent libsodium lz4 xz
libyaml openssl@3

====== Bison 3+

Tebako requires Bison 3+.

On macOS 14, the default Bison version is 2.3, and the Homebrew formula is keg-only, which means that the full path to the Bison binary must be used to utilize the correct version.

Run the following command prior to using Tebako, or add it into your shell profile.

[source,sh]

export PATH="$(brew --prefix bison)/bin:$PATH"

====== jemalloc library build

On macOS, the libdwarfs build script creates an additional jemalloc installation. This is done to satisfy the magic applied by folly during linking but uses a static library.

If the jemalloc library is built within an emulated environment (QEMU, Rosetta, etc.), there are known issues (link:https://github.com/jemalloc/jemalloc/issues/1997[jemalloc issue #1997]) where jemalloc incorrectly detects the number of significant virtual address bits and therefore defines them wrongly (lg-vaddr parameter).

Technically, these issues can be fixed by explicitly setting the --with-lg-vaddr parameter for the jemalloc build. However, we decided not to automate this since we do not feel that we can provide reasonable test coverage.

Instead, our build script accepts the LG_VADDR environment variable and passes it to the jemalloc build as --with-lg-vaddr=${LG_VADDR}.

The LG_VADDR parameter specifies the number of significant virtual address bits, which can vary based on the CPU architecture and emulation status.

This is a simple example script to set LG_VADDR.

NOTE: This is provided for illustration only.

[example]

[source,sh]

#!/bin/bash

Check the CPU architecture

ARCH=$(uname -m)

Check if running under Rosetta 2 emulation

if [[ "$ARCH" == "x86_64" && $(sysctl -n sysctl.proc_translated) == "1" ]]; then echo "Running on Apple Silicon under Rosetta 2 emulation" export LG_VADDR=39 elif [[ "$ARCH" == "arm64" ]]; then echo "Running on Apple Silicon" export LG_VADDR=39 else echo "Running on Intel Silicon" export LG_VADDR=48 fi

echo "Setting lg-vaddr to $LG_VADDR"

====

===== Windows

====== General

The following instructions work for:

Windows 10, 11
Windows Server 2019, 2022

====== Ruby

To run Tebako you need to have Ruby installed. It is simplest to use the Ruby development environment provided by https://rubyinstaller.org[RubyInstaller].

For example, Ruby+Devkit 3.1.4-1.

====== MinGW ucrt64

Enable MinGW ucrt64 and install the necessary packages.

The ridk command originates from the RubyInstaller installation.

[source,sh]

$ ridk enable ucrt64 $ pacman -S git tar bison flex toolchain make cmake boost diffutils libevent double-conversion fmt glog dlfcn gtest autotools ncurses libyaml

== Packaging

[[root-folder-selection]] === Tebako root folder (aka prefix) selection

The Tebako prefix determines the base directory for the Tebako setup.

It is an essential part of configuring how Tebako operates within your system.

The selection of the Tebako prefix follows a specific order of precedence to ensure flexibility and ease of use:

. User-specified prefix: The most direct way to set the root folder is by specifying it through a command-line argument.

. Current Working Directory (PWD): If the prefix option is explicitly set to PWD, Tebako uses the current working directory as Tebako root folder.

. Environment variable (TEBAKO_PREFIX): In the absence of a user-specified option, Tebako looks for an environment variable named TEBAKO_PREFIX. If found, its value is used as the root folder.

. Default value: If no prefix is specified and the TEBAKO_DIR environment variable is not set, Tebako defaults to using a directory named .tebako in the user's home directory.

Path Expansion: Regardless of the method used to set the Tebako prefix, Tebako expands the provided path to an absolute path. This expansion includes resolving relative paths based on the current working directory and expanding user directory shortcuts like ~.

=== Commands

Tebako provides several commands to manage the packaging and deployment process.

==== Press

This command "presses" a Ruby project using the Tebako components built in the Tebako root folder (<tebako-root-folder>).

[NOTE]

The first invocation of the `press` command can take up to an hour as it sets up the packaging environment and collects the required dependencies. Subsequent invocations are much faster.

Upon the next invocation, Tebako will use previously created packaging environment. The press process itself takes minutes.

You can manage setup of packaging environment manually; please refer to description of setup and clean commands below.

[source,sh]

tebako press
-e|--entry-point=
-r|--root=
[-p|--prefix=]
[-R|--Ruby=]
[-o|--output=]
[-l|--log-level=<error|warn|debug|trace>]
[-c|--cwd=]
[-D|--devmode]
[-P|--patchelf]
[-m|--mode=<bundle|both|application|runtime>]
[-u|--ref=]
[-t|--tebafile=]

Where:

--entry-point=<entry-point>:: an executable file (binary executable or script) that shall be started when packaged file is called. This parameter is not required if the operation mode is runtime.

--prefix=<tebako-root-folder>:: the Tebako root folder (see details: <>)

--Ruby=<ruby-version>:: this parameter defines Ruby version that will be packaged (optional, defaults to 3.3.7)

--root=<project-root-folder>:: a folder at the host source file system where project files are located. This parameter is not required if the operation mode is runtime.

--output=<packaged-file-name>:: (optional) the output file name. + On Windows, the output file is automatically appended the .exe extension.

** If the -o option is not specified:

*** in runtime mode, the runtime package is created at tebako-runtime. *** in application mode, the application package is created at <current-folder>/<entry-point-base-name>. *** in bundle mode, the bundled package is created at <current-folder>/<entry-point-base-name>. *** in both mode, the runtime package is created at <current-folder>/<entry-point-base-name>, and the application package is created at <current-folder>/<entry-point-base-name>.tebako.

** If the -o option is specified:

*** in runtime mode, the runtime package is named according to the -o option. *** in application mode, the application package is named according to the -o option. *** in bundle mode, the bundled package is named according to the -o option. *** in both mode, the runtime package is named according to the -o option, and the application package is named according to the -o option with the .tebako extension.

--log-level=<error|warn|debug|trace>:: (optional, defaults to error) logging level for the Tebako built-in memory filesystem driver.

--cwd=<package-current-working-directory>:: (optional) A folder within Tebako memfs where the packaged application will start. This folder should be specified relative to the memfs root. + If not provided, the application will start within the current folder of the host (i.e., at $PWD). + This argument is required because it is not possible to change the directory to a memfs folder until the package is started, as opposed to any host folder that can be set as the current directory before Tebako package invocation. Tebako saves the original working directory in a global Ruby variable $tebako_original_pwd.

--devmode:: flag that activates development mode, in which Tebako's cache and packaging consistency checks are relaxed.

--patchelf:: Allows forward-compatibility of Tebako packages with Linux GNU distributions. + Specifically, this is a flag that removes a reference to the GLIBC_PRIVATE version of libpthread from a Tebako package. This allows Linux GNU packages to run against versions of libpthread that differ from the version used for packaging. + NOTE: This option only works on GNU Linux only. + [example] For example, a package created at Ubuntu 20 system can be used on Ubuntu 22. + NOTE: The feature is exeprimental, we may consider another approach in the future.

--mode=<mode>:: (optional, defaults to bundle) Package output mode, determines whether the runtime and/or application are to be separately packaged.

bundle::: Create a single package bundling both the runtime and application. The output file is named according to the -o option. both::: Create separate packages for the runtime and application. Outputs two separate files: one for the runtime at the location specified at the -o {filename} and one for the application at {filename}.package. application::: Create the application package only. The output file is named according to the -o option. runtime::: Create the runtime package only. The output file is named according to the -o option.

--ref:: (optional) Defaults to tebako-runtime. When a Tebako application package is created on Windows, it is linked against a Tebako runtime file name. The ref parameter allows to specify the name of the runtime file. + NOTE: The --ref option specifies the name of the runtime -- the runtime file can be recreated or changed but not renamed.

--tebafile=<path>:: the tebako configuration file (optional, defaults to $PWD/.tebako.yml). Please refer to the separate section below for description of the tebafile. + NOTE: Development mode is not intended for production use and should only be used during development. + NOTE: --entry-point and --project-root-folder are required parameters and may be provided either via command-line or in the tebafile.

[example]

[source,sh]

tebako press
--root='~/projects/myproject'
--entry=start.rb
--output=/temp/myproject.tebako

====

==== Setup

This command sets up the Tebako packaging environment.

Collects required packages, builds the and creates packaging environment. This is a lengthy task that can take significant time, up to 1 hour.

Tebako supports several configurations at a single system given that their root directories differ and multiple Ruby versions within single configuration

This command is optional, tebako creates packaging environment automatically upon the first invocation of press command.

However, if you plan to use tebako in CI/CD environment with caching it is highly recommended to build cache based on tebako setup output. Building cache based on tebako press may create inconsistent environment upon restore.

[source,sh]

$ tebako setup
[-p|--prefix=]
[-R|--Ruby=]
[-D|--devmode]
[-t|--tebafile=]

Where:

--prefix=<tebako-root-folder>:: the Tebako root folder (see details: <>)

--Ruby=<ruby-version>:: parameter defines package Ruby version (optional). This version is used in conjunction with requirements from the Gemfile as explained below in <>.

--tebafile=<path>:: the tebako configuration file (optional, defaults to $PWD/.tebako.yml). Please refer to the separate section below for tebafile description.

--devmode:: flag activates development mode, in which Tebako's cache and packaging consistency checks are relaxed. Please note that this mode is not intended for production use and should only be used during development.

==== Clean

This command cleans up all Tebako artifacts in the specified prefix directory.

NOTE: These artifacts are created by the setup and press commands. Normally you do not need to do it since tebako packager optimizes artifacts lifecycle on its own.

[source,sh]

$ tebako clean
[-p|--prefix=]
[-t|--tebafile=]

Where:

--prefix=<tebako-root-folder>:: the Tebako root folder (see details: <>)

--tebafile=<path>:: the tebako configuration file (optional, defaults to $PWD/.tebako.yml). Please refer to the separate section below for tebafile description.

[example]

[source,sh]

tebako clean --prefix='~/.tebako'

====

==== Clean Ruby

This command cleans up only the Ruby artifacts from the specified prefix directory.

NOTE: These artifacts are created by the setup and press commands. Normally you do not need to do it, since Tebako packager optimizes artifacts lifecycle on its own.

NOTE: Compiled DwarFS libraries are not cleaned.

[source,sh]

$ tebako clean_ruby [-p|--prefix=]
[-R|--Ruby=]
[-t|--tebafile=]

Where:

--prefix=<tebako-root-folder>:: the Tebako setup folder (optional, defaults to current folder)

--Ruby=<ruby-version>:: (optional, cleans all versions by default) defines Ruby version that will be cleaned.

--tebafile=<path>:: the tebako configuration file (optional, defaults to $PWD/.tebako.yml). Please refer to the separate section below for tebafile description.

[example]

[source,sh]

tebako clean_ruby --prefix='~/.tebako'

====

==== Build script hash

This command outputs a hash value for the Tebako build script, which can be used as a cache key in CI/CD pipelines.

[source,sh]

$ tebako hash

=== Tebako configuration file

It is possible to provide all or some options for the tebako {setup | press | clean | clean_ruby} commands via a Tebako configuration file ('tebafile').

Tebafile is a YAML file with a single key options. The options are the same as long names for the command line.

For example, for the prefix option:

[source]

-p|--prefix=

the key in the YAML file would be 'prefix'.

.Example tebafile that sets values for prefix and Ruby options [example]

This is an example tebafile that sets values for prefix and Ruby options:

[source,yaml]

options: prefix: /tmp/tebako Ruby: 3.2.4

====

=== Options prefernce order

Tebako supports several methods to set options. The table below show preference order and limitations for specific options. samller order means higher proirity.

|===

=== Exit codes

The Tebako CLI exits with different exit codes to indicate the status of the operation. The following table lists the possible exit codes and their meanings.

.Tebako CLI exit codes [cols="a,a"] |=== | Code | Condition

| 0 | No error | 1 | Invalid command line | 101 | 'tebako setup' configure step failed | 102 | 'tebako setup' build step failed | 103 | 'tebako press' configure step failed | 104 | 'tebako press' build step failed | 105 | Failed to map MSys path to Windows | 106 | Entry point does not exist or is not accessible | 107 | Project root does not exist or is not accessible | 108 | Package working directory does not exist | 109 | Invalid Ruby version format | 110 | Ruby version is not supported | 111 | Ruby version is not supported on Windows | 112 | OS is not supported | 113 | Path to root shall be absolute. Relative path is not allowed | 114 | Entry point is not within the project root | 115 | Failed to load Gemfile | 116 | Ruby version does not satify Gemfile requirements |===

[[ruby-and-bundler-versions]] == Ruby and bundler versions selection

During packaging tebako creates its own Ruby execution environment that is independent from the host Ruby environment. The version of Ruby that is used in the package is defined by the combination of the --Ruby option and ruby requirement specified in Gemfile.

.Package Ruby version selection rules [cols="3", options="header"] |=== | Gemfile requirement | --Ruby option given | --Ruby option absent

| Specified | The version specified by --Ruby option if it is supported and satisfies Gemfile requirement; error otherwise | The minimal supported Ruby version that satisfies Gemfile requirement; error otherwise | Not specified | The version specified by --Ruby option if it is supported; error otherwise | Default Tebako Ruby version (3.3.7).

|===

When bundler is used to package the application, the version of bundler that is used is determined by the Ruby version and the bundler version requirements specified in the Gemfile.

.Package bundler version selection rules [cols="3", options="header"] |=== | Gemfile requirement | Ruby version 2.7.8, 3.0.7 | Ruby version 3.1.x or above

| Specified | The maximum version that meets Gemfile requirement if it is >= 2.4.22 ; error otherwise | The maximum version that meets Gemfile requirement if it is >= 2.4.22 ; error otherwise | Not specified | 2.4.22 | Default bundler version for this version of Ruby.

|===

== Packaging scenarios

Depending on the configuration files that are present in the root project folder, the Tebako Ruby packager supports different packaging scenarios.

These scenarios differ in what files are packaged and where the entry point is located.

Here is a summary of the scenarios:

| 1 | Simple ruby script | Copy <project-root> with all sub-folders to packaged filesystem | <mount_point>/local/<entry_point base name> | No | No | No

| 2 | Packaged gem | Install the gem with gem install to packaged filesystem | <mount_point>/bin/<entry_point base name> (i.e., binstub is expected) | No | No | One

| 3 | Gem source, no bundler | . Build the gem using gem build command at the host . Install it with gem install to packaged filesystem

| <mount_point>/bin/<entry_point base name> (i.e., binstub is expected) | One | No | Any

| 4 | Gem source, bundler | . Collect dependencies at the host with bundle install . Build the gem using gem build command . Install it with gem install to packaged file system

| <mount_point>/bin/<entry_point base name> (i.e., binstub is expected) | One | One | Any

| Error

Error: Two or more `*.gem` files present
-
No
No
Two or more

| Error

Error: Two or more `*.gemspec` files present
-
Two or more
Any
Any

|===

These scenarios determine how the project is packaged and where the entry point is located within the packaged filesystem.

== Run-time options

=== General

Generally Tebako package passes command line options to the packaged application.

[example]

For example, if the package was created with the following command

[source,sh]

tebako press
--root='~/projects/myproject'
--entry=start.rb
--output=/temp/myproject.tebako

running

[source,sh]

/temp/myproject.tebako --option --parameter value

will be translated by Tebako bootstrap code to

[source,sh]

myproject --option --parameter value

====

However there are several command-line parameters that are intercepted processed by Tebako bootstrap code as described below.

=== Creating a bundle package

Tebako provides a method to create and manage a bundle package, which simplifies the process of packaging dependencies along with the application.

[source,sh]

$ tebako bundle create -o -r [-R ]

To run the bundle package, use the following command:

[source,sh]

$

[example]

[source,sh]

$ tebako bundle create -o myproject-bundle -r ~/projects/myproject -R 3.4.1

=> creates `myproject-bundle`, a bundle package for Ruby 3.4.1

$ myproject-bundle

=> runs the bundle package

====

=== Creating separate runtime and application packages

Tebako allows creating separate runtime and application packages that can be used to run a Tebako application package.

[source,sh]

$ tebako press -m both
-o
-e
-r
[-R ]

The resulting packages will be generated in the current directory as:

Tebako runtime package at the -o location, i.e. <tebako-runtime-package>.
Tebako application package. The name of the application package will be the same as the runtime package with the .package extension.

=== Creating a Tebako runtime package

Tebako allows creating a Tebako runtime package that can be used to run a Tebako application package.

[source,sh]

$ tebako press -m runtime -o [-R ]

[example]

[source,sh]

$ tebako press -o tebako-ruby-3.4.1 -R 3.4.1

=> creates `tebako-ruby-3.4.1`, a Tebako runtime package for Ruby 3.4.1

====

=== Creating a Tebako application package

Tebako allows creating a Tebako application package that can be run with a Tebako runtime package.

[source,sh]

$ tebako press -m application
-o
-e
-r
[-R ]

[example]

[source,sh]

$ tebako press -m application
-o tebako-application-package
-e start.rb
-r ~/projects/myproject
-R 3.4.1

=> creates `tebako-application-package`, a Tebako application package for Ruby 3.4.1

====

=== Running Tebako application using a Tebako runtime (--tebako-run option)

The Tebako application package can be executed by a Tebako runtime package.

[source,sh]

$ --tebako-run []

Where,

<tebako-runtime-package>:: The Tebako runtime package file created using in runtime or both mode.

<tebako-application-package>:: The Tebako application package created in application or both mode.

[example]

Given a Ruby application at hello.rb:

[source,Ruby]

puts "Hello, #{ARGV[0]}!"

Create separate runtime and application packages:

[source,sh]

tebako press -m runtime -o tebako-runtime-package tebako press -m application -o tebako-application-package -e hello.rb -r test

Run the application using the pre-packaged runtime:

[source,sh]

tebako-runtime-package --tebako-run tebako-application-package Maxim

The expected output is:

[source]

Hello, Maxim!

====

=== Image extraction (--tebako-extract option)

Tebako provides an option to extract its DwarFS filesystem from a package to a local folder for verification or execution.

[source,sh]

$ --tebako-extract []

Where,

<root folder for extracted filesystem>:: The root folder for the extracted filesystem (optional, defaults to source_filesystem)

[example]

Extracting Tebako content from the metanorma package:

[source,sh]

metanorma --tebako-extract temp-image

====

The --tebako-extract option actually runs the following Ruby script:

[source,ruby]

require 'fileutils' FileUtils.copy_entry '', ARGV[2] || 'source_filesystem'

=== Mounting host folder to Tebako memfs (--tebako-mount option)

Some programs unconditionally use folders located under the application root, and when processed by Tebako or similar tools, these folders are included in the packaging.

[example]

Rails, for example, does not provide a configuration option to change where it expects the tmp folder to be.

The location is hardcoded in multiple places within the Rails codebase, residing under the application root, and as a result, it gets included in the read-only Tebako memfs.

Although patches have been proposed (e.g., https://github.com/rails/rails/issues/39583), there is currently no way to change the paths for temporary files, caches, and sockets.

To address this inevitable limitation for Ruby applications, Tebako provides an option to mount a host folder to the memfs tree.

When using Tebako, consider the packaging scenario mentioned above, as it defines the layout of the application tree.

The --tebako-extract option may be useful for understanding the placement of files and folders.

[example]

The following command starts a rails.tebako package with $PWD/tmp mounted as local/tmp in the memfs.

Any remaining command-line parameters are passed to the application.

[source,sh]

rails.tebako --tebako-mount local/tmp:$PWD/tmp server

====

The --tebako-mount option has the following syntax:

[source,sh]

--tebako-mount :

The --tebako-mount option can be repeated multiple times to mount more than one object. The memfs path is relative to the memfs root, and it is recommended to use absolute paths for host objects. Both directories and files can be mounted in this way. Tebako allows overlaying existing memfs objects, so there are no significant limitations.

== Trivia: origin of name

"tamatebako" (玉手箱) is the treasure box given to Urashima Taro in the Ryugu, for which he was asked not to open if he wished to return. He opened the box upon the shock from his return that three hundred years has passed. Apparently what was stored in the box was his age.

This packager was made to store Ruby and its gems, and therefore named after the said treasure box (storing gems inside a treasure box).

Since "tamatebako" is rather long for the non-Japanese speaker, we use "tebako" (手箱, also "tehako") instead, the generic term for a personal box.

== Contributing

We welcome contributions! Please see our contributing guidelines for more information.

== License

Tebako is released under the BSD 2-Clause License. See the LICENSE file for details.

gittech. site

Tebako: An executable packager (for Ruby programs)

[NOTE]

We saw clang compiler segmentation fault when packaging of very large projects with XCode 14.3.1. This issue is not reproducible with XCode 15.0.1 or higher.

[source,sh]

docker pull ghcr.io/tamatebako/tebako-:latest

[source,sh]

docker run -v :/mnt/w -t ghcr.io/tamatebako/tebako-:latest tebako {command} {parameters}

[source,sh]

docker run -v :/mnt/w -t ghcr.io/tamatebako/tebako-:latest tebako press

[example]

[source,sh]

docker run -v $PWD:/mnt/w -t ghcr.io/tamatebako/tebako-ubuntu-20.04:latest tebako press --root=/mnt/w/fontist --entry-point=fontist --output=/mnt/w/fontist-package --Ruby=3.2.4

[source,sh]

docker run -it --rm -v :/mnt/w ghcr.io/tamatebako/tebako-:latest bash

[source,sh]

tebako press

[example]

[source,sh]

Inside the container:

$ tebako press --root=/mnt/w/fontist --entry-point=fontist --output=/mnt/w/fontist-package --Ruby=3.2.4

[source,sh]

$ gem install tebako

[source,sh]

apt install -y gcc-10 g++-10 update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-10 10 update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-10 10

[source,sh]

apt install -y clang-12 update-alternatives --install /usr/bin/clang clang /usr/bin/clang-12 150 update-alternatives --install /usr/bin/clang++ clang++ /usr/bin/clang++-12 150

.Installing CMake 3.20+ [source,sh]

apt-get remove --purge --auto-remove cmake apt-get update apt-get install -y software-properties-common lsb-release curl apt-get clean all curl https://apt.kitware.com/kitware-archive.sh | bash apt-get install cmake

[source,sh]

[source,sh]

[source,sh]

$ brew bundle

[source,sh]

$ brew update $ brew install gnu-sed bash pkg-config bison flex binutils libffi gdbm zlib ncurses double-conversion boost jemalloc fmt glog libevent libsodium lz4 xz libyaml openssl@3

[source,sh]

export PATH="$(brew --prefix bison)/bin:$PATH"

[example]

[source,sh]

Check the CPU architecture

Check if running under Rosetta 2 emulation

echo "Setting lg-vaddr to $LG_VADDR"

[source,sh]

[NOTE]

The first invocation of the press command can take up to an hour as it sets up the packaging environment and collects the required dependencies. Subsequent invocations are much faster.

[source,sh]

tebako press -e|--entry-point= -r|--root= [-p|--prefix=] [-R|--Ruby=] [-o|--output=] [-l|--log-level=<error|warn|debug|trace>] [-c|--cwd=] [-D|--devmode] [-P|--patchelf] [-m|--mode=<bundle|both|application|runtime>] [-u|--ref=] [-t|--tebafile=]

[example]

[source,sh]

tebako press --root='~/projects/myproject' --entry=start.rb --output=/temp/myproject.tebako

[source,sh]

$ tebako setup [-p|--prefix=] [-R|--Ruby=] [-D|--devmode] [-t|--tebafile=]

[source,sh]

$ tebako clean [-p|--prefix=] [-t|--tebafile=]

[example]

[source,sh]

tebako clean --prefix='~/.tebako'

[source,sh]

[example]

[source,sh]

tebako clean_ruby --prefix='~/.tebako'

[source,sh]

$ tebako hash

[source]

-p|--prefix=

.Example tebafile that sets values for prefix and Ruby options [example]

[source,yaml]

options: prefix: /tmp/tebako Ruby: 3.2.4

[example]

[source,sh]

tebako press --root='~/projects/myproject' --entry=start.rb --output=/temp/myproject.tebako

[source,sh]

/temp/myproject.tebako --option --parameter value

[source,sh]

myproject --option --parameter value

[source,sh]

$ tebako bundle create -o -r [-R ]

[source,sh]

$

[example]

We saw `clang` compiler segmentation fault when packaging of very large projects with XCode 14.3.1. This issue is not reproducible with XCode 15.0.1 or higher.

docker run -v :/mnt/w
-t ghcr.io/tamatebako/tebako-:latest
tebako {command} {parameters}

docker run -v :/mnt/w
-t ghcr.io/tamatebako/tebako-:latest
tebako press

docker run -v $PWD:/mnt/w
-t ghcr.io/tamatebako/tebako-ubuntu-20.04:latest
tebako press --root=/mnt/w/fontist --entry-point=fontist --output=/mnt/w/fontist-package --Ruby=3.2.4

docker run -it --rm -v :/mnt/w
ghcr.io/tamatebako/tebako-:latest bash

$ brew update $ brew install gnu-sed bash pkg-config bison flex binutils libffi gdbm zlib
ncurses double-conversion boost jemalloc fmt glog libevent libsodium lz4 xz
libyaml openssl@3

The first invocation of the `press` command can take up to an hour as it sets up the packaging environment and collects the required dependencies. Subsequent invocations are much faster.

tebako press
-e|--entry-point=
-r|--root=
[-p|--prefix=]
[-R|--Ruby=]
[-o|--output=]
[-l|--log-level=<error|warn|debug|trace>]
[-c|--cwd=]
[-D|--devmode]
[-P|--patchelf]
[-m|--mode=<bundle|both|application|runtime>]
[-u|--ref=]
[-t|--tebafile=]

tebako press
--root='~/projects/myproject'
--entry=start.rb
--output=/temp/myproject.tebako

$ tebako setup
[-p|--prefix=]
[-R|--Ruby=]
[-D|--devmode]
[-t|--tebafile=]

$ tebako clean
[-p|--prefix=]
[-t|--tebafile=]

tebako press
--root='~/projects/myproject'
--entry=start.rb
--output=/temp/myproject.tebako

=> creates `myproject-bundle`, a bundle package for Ruby 3.4.1

$ tebako press -m both
-o
-e
-r
[-R ]

=> creates `tebako-ruby-3.4.1`, a Tebako runtime package for Ruby 3.4.1

=> creates `tebako-application-package`, a Tebako application package for Ruby 3.4.1