Yarhl, A format ResearcH Library

Yarhl is a set of libraries that helps to implement and convert file formats It empowers you with...

♻️ ... APIs to easily convert between custom formats.
📚 ... guidelines to implement and test custom format converters.
🔢 ... advance binary and text reading / writing, encoding and serialization.
📃 ... standard formats implementation like PO for translations.
📂 ... virtual file system to unpack and pack containers efficiently.
🔌... plugin API to find formats and converters in .NET assemblies.

Usage

The project has the following .NET libraries (NuGet packages via nuget.org). The libraries only support .NET LTS versions: .NET 6.0 and .NET 8.0.

- Yarhl.FileFormat: format conversion APIs.
- Yarhl.FileSystem: virtual file system.
- Yarhl.IO: streams, binary and text reading / writing.
- Yarhl.Media.Text: translation formats and converters (Po), table replacer.
- Yarhl.Media.Text.Encoding: euc-jp and token-escaped encodings.
- Yarhl.Plugins: load nearby .NET assemblies and find type implementations.
- Yarhl.Plugins.FileFormat: find formats and converters from loaded assemblies.

Note

Are you planning to try a preview version?
Check-out the GitHub project readme for details on how to setup the NuGet preview feed for SceneGate projects.

Quick demo

Yarhl allows you to work with different file formats with an unified API for conversion into binary (serialization / deserialization). Let's try to create a new translatable file format PO from Yarhl.Media.Text and save it into disk.

// Let's create a new PO format
var metadata = new PoHeader("software1", "SceneGate", "es-ES");
Po translatableContent = new Po(metadata);

// Now let's add one entry.
var entry1 = new PoEntry("Hello world!");
entry1.Translated = "¡Hola mundo!";
translatableContent.Add(entry1);

// Finally let's save the format into a file on disk
var serializer = new Po2Binary();
using BinaryFormat binaryPo = serializer.Convert(translatableContent);
binaryPo.Stream.WriteTo("strings.po");

It's frequent to find formats that are containers. Yarhl allows to have a virtual file system to work with its content without having to extract it into disk (saving space and time). For instance, let's open a game file from Nintendo DS that contains thousand of files. Then we will navigate through its files, unpacking, decompressing and finally converting one file into PO. We can use the following libraries for this task:

Ekona: support of NDS game file system.
LayTea: support of formats from Professor Layton games.

// 1. Read the game file system from a file (deserialize)
using Node game = NodeFactory.FromFile(gameFilePath, FileOpenMode.Read)
    .TransformWith<Binary2NitroRom>();

// 2. Navigate to the container that has our text file and unpack it.
Node msgNode = Navigator.SearchNode(game, "data/ll/common/ll_common.darc")
    .TransformWith<BinaryDarc2Container>() // binary -> file system (container)
    .Children[2]                           // text file is the third file
    .TransformWith<DencDecompression>();   // the file is compressed with LZSS

// 3. Convert its proprietary binary format into industry-standard translation format PO.
//    As it's a big text file, the converter splits the content into different files.
msgNode.TransformWith<Binary2MessageCollection>()
    .TransformWith<MessageCollection2PoContainer, LondonLifeRegion>(LondonLifeRegion.Usa);

foreach (var children in msgNode.Children) {
    // 4. Convert the PO into a binary format (serialize) and write to disk
    children.TransformWith<Po2Binary>()
        .Stream.WriteTo(children.Name + ".po");
}

Showcase

Some cool projects built with Yarhl:

Texim: experimental API for image file formats.
Ekona: support Nintendo DS file formats.
Lemon: support Nintendo 3DS file formats.
LayTea: modding tools for Professor Layton games.
Attack of Friday Monsters tools: modding tools for Attack of the Friday Monsters game.
Metatron: translation framework for Shin Megami Tensei saga games.

License