docs: reorganize macro docs, add "tasks" concept

This commit is contained in:
Travis Abendshien
2025-08-14 15:45:40 -07:00
parent 119b964b16
commit 1110f64ff5
2 changed files with 228 additions and 154 deletions

View File

@@ -28,13 +28,11 @@ triggers = ["on_new_entry"]
## Actions
Actions are the general task that a macro will perform. They are represented by TOML tables and must have a unique name in your macro file, but the name itself has no importance to the macro. The root table will contain general configuration info about your action, and nested tables (e.g. `[action.task]`) will further define the specifics of your action.
Actions are broad categories of operations that your macro will perform. They are represented by TOML tables and must have a unique name in your macro file, but the name itself has no importance to the macro. A single macro file can contain multiple actions and each action can contain multiple tasks.
```toml
[newgrounds]
```
An action table with a name of your choosing (e.g. `[action]`) will contain the general configuration for your action, and nested task tables (e.g. `[action.task]`) will define the specifics of your action's tasks.
Action tables must have an `action` key with one of the following values:
Action tables must have an `action` key with one of the following valid action values:
- [`import_data`](#import-data): Import data from a supported external source.
- [`add_data`](#add-data): Add data declared inside the macro file.
@@ -44,15 +42,33 @@ Action tables must have an `action` key with one of the following values:
action = "import_data"
```
Most of the configuration of actions comes at the [task configuration](#task-configuration) level. This is where you will build out exactly how your action will translate data and instructions into results for your TagStudio library.
---
### Add Data
The `add_data` action lets you add data to a [file entry](entries.md) given one or more conditional statements. Unlike the [`import_data`](#import-data) action, the `add_data` action adds data declared in the macro itself rather than importing it form a source external to the macro.
Compatible Keys:
- [`source_filters`](#source_filters)
- [`value`](#value)
---
### Import Data
The `import_data` action allows you to import external data into your TagStudio library in the form of [tags](tags.md) and [fields](fields.md). While some sources need explicit support (e.g. ID3, EXIF) generic sources such as JSON sidecar files can leverage wide array of data shaping options that allow the underlying data structure to be abstracted from TagStudio's internal data structures. This macro pairs very well with tools such as [gallery-dl](https://github.com/mikf/gallery-dl).
The `import_data` action allows you to import external data into your TagStudio library in the form of [tags](tags.md) and [fields](fields.md). While some sources need explicit support (e.g. ID3, EXIF) generic sources such as JSON sidecar files can leverage a wide array of data shaping options that allow the underlying data structure to be abstracted from TagStudio's internal data structures. This macro pairs very well with tools that download sidecar files for data such as [gallery-dl](https://github.com/mikf/gallery-dl).
If you're importing from an object or table-like source (i.e. JSON), you'll need to create a nested table with the name format of "`[action.task]`" and provide a [`key`](#key) field filled with the name of the targeted source key. In this case the "task" name does not matter as long as it doesn't conflict with one of the built-in names (i.e. "`map`", "`template`, "`inverse_map`").
Compatible Keys:
- [`key`](#key)
- [`source_location`](#source_location)
- [`source_format`](#source_format)
- [`is_embedded`](#is_embedded)
If you're importing from an object-like source (e.g. JSON), you'll need to create a nested task table with the format `[action.task]` and provide a [`key`](#key) field filled with the name of the targeted source key. In this case the task name does not matter as long as it doesn't conflict with one of the built-in task names (i.e. "`map`", "`inverse_map`, "`template`").
<!-- prettier-ignore -->
=== "Importable JSON Data"
@@ -73,11 +89,129 @@ If you're importing from an object or table-like source (i.e. JSON), you'll need
Inside the new table we can now declare additional information about the native data formats and how they should be imported into TagStudio.
---
### Action Configuration
#### Key
#### `source_format`
The `key` key is used in conjunction with the [`import_data`](#import-data) action to specify the object/table key to target in your import data. If you're targeting a nested object, separate the names of the keys with a dot.
<!-- prettier-ignore -->
!!! note ""
Compatible Actions: [`import_data`](#import-data)
The `source_format` key is used to declare what type of source data will be imported from.
```toml
[newgrounds]
action = "import_data"
source_format = "json"
```
- `exif`: Embedded EXIF metadata
- `id3`: Embedded ID3 metadata
- `json`: A JSON formatted file
- `text`: A plaintext file
- `xml`: An XML formatted file
- `xmp`: Embedded XMP metadata or an XMP sidecar file
---
#### `source_location`
<!-- prettier-ignore -->
!!! note ""
Compatible Actions: [`import_data`](#import-data)
The `source_location` key is used to declare where the metadata should be imported from. This can be a relative or absolute path, and can reference the targeted filename with the `{filename}` placeholder.
```toml
[newgrounds]
action = "import_data"
source_format = "json"
source_location = "{filename}.json" # Relative sidecar file
```
<!-- - `absolute`: An absolute file location
- `embedded`: Data that's embedded within the targeted file
- `sidecar`: A sidecar file with a relative file location -->
---
#### `is_embedded`
<!-- prettier-ignore -->
!!! note ""
Compatible Actions: [`import_data`](#import-data)
If targeting embedded data, add the `is_embedded` key and set it to `true`. If no `source_location` is used then the file this macro is targeting will be used as a source.
```toml
[newgrounds]
action = "import_data"
source_format = "id3"
is_embedded = true
```
---
#### `source_filters`
<!-- prettier-ignore -->
!!! note ""
Compatible Actions: [`add_data`](#add-data), [`import_data`](#import-data)
`source_filters` are used to declare a glob list of files that are able to be targeted by this action. An entry filepath only needs to fall under one of the given source filters in order for the macro to continue. If not, then the macro will be skipped for this file entry.
<!-- prettier-ignore -->
=== "import_data"
```toml
[newgrounds]
action = "import_data"
source_format = "json"
source_location = "{filename}.json"
source_filters = ["**/Newgrounds/**"]
```
=== "add_data"
```toml
[animated]
action = "add_data"
source_filters = ["**/*.gif", "**/*.apng"]
```
<!-- ### Source Types
The `source_type` key allows for the explicit declaration of the type and/or format of the source data. When this key is omitted, TagStudio will default to the data type that makes the most sense for the destination [TagStudio type](#tagstudio-types).
- `string`: A character string (text)
- `integer`: An integer
- `float`: A floating point number
- `url`: A string with a special URL formatting pass
- [`ISO8601`](https://en.wikipedia.org/wiki/ISO_8601) A standard datetime format
- `list:string`: List of strings (text)
- `list:integer`: List of integers
- `list:float`: List of floating point numbers -->
---
## Task Configuration
An [action's](#actions) tasks need to be configured using the built-in keys available to each action. These keys may be specific to certain actions, required or optional, or expect other specific formatting. The actions section will list each action's available keys, and the following list of keys will likewise list which actions they are compatible with along with any other rules.
Along with generally defining your own custom tasks, there are a few built-in tasks that have reserved names and offer extra functionality on top of your own tasks. These currently include:
- [`.inverse-map`](#many-to-1-inverse-map) (Inverse Tag Maps)
- [`.map`](#manual-tag-mapping) (Tag Maps)
- [`.template`](#templates) (Templates)
---
### `key`
<!-- prettier-ignore -->
!!! note ""
Compatible Actions: [`import_data`](#import-data)
The `key` key is used to specify the object key to target in your data source. If you're targeting a nested object, separate the names of the keys with a dot.
```toml
[artstation]
@@ -91,7 +225,7 @@ key="mediums.name" # Nested key
ts_type = "tags"
```
When importing from the same key multiple times, you have the option to either choose different names for your "task" tables or use the same name with these tables wrapped in an extra pair of brackets.
When importing from the same key multiple times, you have the option to either choose different names for your task tables or use the same name with these tables wrapped in an extra pair of brackets.
<!-- prettier-ignore -->
=== "Single Import"
@@ -133,108 +267,13 @@ When importing from the same key multiple times, you have the option to either c
name = "Artist"
```
#### Source Format
---
The `source_format` key is used in conjunction with the [`import_data`](#import-data) action to declare what type of source data will be imported from.
```toml
[newgrounds]
action = "import_data"
source_format = "json"
```
- `exif`: Embedded EXIF metadata
- `id3`: Embedded ID3 metadata
- `json`: A JSON formatted file
- `text`: A plain text file
- `xml`: An XML formatted file
- `xmp`: Embedded XMP metadata or an XMP sidecar file
#### Source Location
The `source_location` key is used in conjunction with the `import_data` key to declare where the metadata should be imported from. This can be a relative or absolute path, and can reference the targeted filename with the `{filename}` placeholder.
```toml
[newgrounds]
action = "import_data"
source_format = "json"
source_location = "{filename}.json" # Relative sidecar file
```
<!-- - `absolute`: An absolute file location
- `embedded`: Data that's embedded within the targeted file
- `sidecar`: A sidecar file with a relative file location -->
#### Embedded Metadata
If targeting embedded data, add the `is_embedded` key and set it to `true`. If no `source_location` is used then the file this macro is targeting will be used as a source.
```toml
[newgrounds]
action = "import_data"
source_format = "id3"
is_embedded = true
```
#### Source Filters
`source_filters` are used to declare a glob list of files that are able to be targeted by this action. An entry filepath only needs to fall under one of the given source filters in order for the macro to continue. If not, then the macro will be skipped for this file entry.
### `ts_type`
<!-- prettier-ignore -->
=== "import_data"
```toml
[newgrounds]
action = "import_data"
source_format = "json"
source_location = "{filename}.json"
source_filters = ["**/Newgrounds/**"]
```
=== "add_data"
```toml
[animated]
action = "add_data"
source_filters = ["**/*.gif", "**/*.apng"]
```
#### Value
The `value` key is specifically used with the [`add_data`](#add-data) action to define what value should be added to the file entry.
<!-- prettier-ignore -->
=== "Title Field"
```toml
[animated]
action = "add_data"
source_filters = ["**/*.gif", "**/*.apng"]
[animated.title]
ts_type = "text_line"
name = "Title"
value = "Animated Image"
```
=== "Tags"
```toml
[animated]
action = "add_data"
source_filters = ["**/*.gif", "**/*.apng"]
[animated.tags]
ts_type = "tags"
value = ["Animated"]
```
<!-- #### Source Types
The `source_type` key allows for the explicit declaration of the type and/or format of the source data. When this key is omitted, TagStudio will default to the data type that makes the most sense for the destination [TagStudio type](#tagstudio-types).
- `string`: A character string (text)
- `integer`: An integer
- `float`: A floating point number
- `url`: A string with a special URL formatting pass
- [`ISO8601`](https://en.wikipedia.org/wiki/ISO_8601) A standard datetime format
- `list:string`: List of strings (text)
- `list:integer`: List of integers
- `list:float`: List of floating point numbers -->
#### TagStudio Types
!!! note ""
Compatible Actions: [`add_data`](#add-data), [`import_data`](#import-data)
The required `ts_type` key defines the destination data format inside TagStudio itself. This can be [tags](tags.md) or any [field](fields.md) type.
@@ -260,9 +299,7 @@ The required `ts_type` key defines the destination data format inside TagStudio
ts_type = "tags"
```
### Field Specific Keys
#### Name
#### Field Specific Keys
`name`: The name of the field to import into.
@@ -286,12 +323,10 @@ The required `ts_type` key defines the destination data format inside TagStudio
!!! note
As of writing (v9.5.3) TagStudio fields still do not allow for custom names. The macro system is designed to be forward-thinking with this feature in mind, however only existing TagStudio field names are currently considered valid. Any invalid field names will default to the "Notes" field.
### Tag Specific Keys
#### Tag Specific Keys
Since TagStudio tags are more complex than other traditional tag formats, there are several options for fine-tuning how tags should be imported.
#### Delimiter
`delimiter`: The delimiter between string tags to use.
<!-- prettier-ignore -->
@@ -308,36 +343,6 @@ Since TagStudio tags are more complex than other traditional tag formats, there
delimiter = "\n"
```
#### Prefix
`prefix`: An optional prefix to remove.
<!-- prettier-ignore -->
!!! example
Given a list of tags such as `["#tag1", "#tag2", "#tag3"]`, you may wish to remove the "`#`" prefix.
```toml
[instagram.tags]
ts_type = "tags"
prefix = "#"
```
#### Strict
`strict`: Determines what [names](tags.md#naming-tags) of the TagStudio tags should be used to compare against the source data when matching.
- `true`: Only match against the TagStudio tag [name](tags.md#name) field.
- `false` (Default): Match against any TagStudio tag name field including [shorthands](tags.md#shorthand), [aliases](tags.md#aliases), and the [disambiguation name](tags.md#disambiguation).
#### Use Context
`use_context`: Determines if TagStudio should use context clues from other source tags to provide more accurate tag matches.
- `true` (Default): Use context clue matching (slower, less ambiguous).
- `false`: Ignore surrounding source tags (faster, more ambiguous).
#### On Missing
`on_missing`: Determines the behavior of how to react to source tags with no match in the library.
- `"prompt"`: Ask the user if they wish to create, skip, or manually choose an existing tag.
@@ -352,9 +357,65 @@ use_context = true
on_missing = "create"
```
`prefix`: An optional prefix to remove.
<!-- prettier-ignore -->
!!! example
Given a list of tags such as `["#tag1", "#tag2", "#tag3"]`, you may wish to remove the "`#`" prefix.
```toml
[instagram.tags]
ts_type = "tags"
prefix = "#"
```
`strict`: A flag that determines what [names](tags.md#naming-tags) of the TagStudio tags should be used to compare against the source data when matching.
- `true`: Only match against the TagStudio tag [name](tags.md#name) field.
- `false` (Default): Match against any TagStudio tag name field including [shorthands](tags.md#shorthand), [aliases](tags.md#aliases), and the [disambiguation name](tags.md#disambiguation).
`use_context`: A flag that determines if TagStudio should use context clues from other source tags to provide more accurate tag matches.
- `true` (Default): Use context clue matching (slower, less ambiguous).
- `false`: Ignore surrounding source tags (faster, more ambiguous).
\*\*
---
### `value`
<!-- prettier-ignore -->
!!! note ""
Compatible Actions: [`add_data`](#add-data)
The `value` key is use specifically with the [`add_data`](#add-data) action to define what value should be added to the file entry.
<!-- prettier-ignore -->
=== "Title Field"
```toml
[animated]
action = "add_data"
source_filters = ["**/*.gif", "**/*.apng"]
[animated.title]
ts_type = "text_line"
name = "Title"
value = "Animated Image"
```
=== "Tags"
```toml
[animated]
action = "add_data"
source_filters = ["**/*.gif", "**/*.apng"]
[animated.tags]
ts_type = "tags"
value = ["Animated"]
```
---
### Manual Tag Mapping
If the results from the standard tag matching system aren't good enough to properly import specific source data into your TagStudio library, you have the option to manually specify mappings between source and destination tags. A table with the `.map` or `.inverse_map` suffixes will be used to map tags in the nearest scope.
If the automatic tag matching system isn't enough to import tags the way you'd like, you can manually specify mappings between source and destination tags. Tables with the `.map` or `.inverse_map` task suffixes will be used to map tags in the nearest scope.
<!-- prettier-ignore -->
=== "Global Scope"
@@ -364,21 +425,23 @@ If the results from the standard tag matching system aren't good enough to prope
```
=== "Action Scope"
```toml
# Applies to all tag keys in the "newgrounds" action
# Applies to all tasks in the "newgrounds" action
[newgrounds.map]
```
=== "Key Scope"
```toml
# Only applies to tags within the "ratings" key inside the "newgrounds" action
# Only applies to the "ratings" task inside the "newgrounds" action
[newgrounds.ratings.map]
```
- `map`: Used for [1 to 0](#1-to-0-ignore-matches), [1 to 1](#1-to-1), and [1 to Many](#1-to-many) mappings.
- `inverse_map`: Used for [Many to 1](#many-to-1-inverse-map) mappings.
- `map`: Used for "[1 to 0](#1-to-0-ignore-matches)", "[1 to 1](#1-to-1)", and "[1 to many](#1-to-many)" mappings
- `inverse_map`: Used for "[many to 1](#many-to-1-inverse-map)" mappings
---
#### 1 to 0 (Ignore Matches)
By mapping the key of the source tag name to an empty string, you can ignore that tag when matching with your own library. This is useful if you're importing from a source that uses tags you don't wish to use or create inside your own libraries.
By mapping the key of the source tag name to an empty string, you can ignore that tag when matching with your own tags. This is useful if you're importing from a source that uses tags you don't wish to use or create inside your own libraries.
```toml
[newgrounds.tags.map]
@@ -386,9 +449,11 @@ By mapping the key of the source tag name to an empty string, you can ignore tha
favorite = ""
```
---
#### 1 to 1
By mapping the key of the source tag name to the name of one of your TagStudio tags, you can directly specify a destination tag while bypassing the matching algorithm.
By mapping the key or quoted string of a source tag to one of your TagStudio tags, you can directly specify a destination tag while bypassing the matching algorithm.
<!-- prettier-ignore -->
!!! tip
@@ -398,11 +463,14 @@ By mapping the key of the source tag name to the name of one of your TagStudio t
[newgrounds.tags.map]
# Source Tag Name = TagStudio Tag Name
colored_pencil = "Drawing"
"Colored Pencil" = "Drawing"
```
---
#### 1 to Many
By mapping the key of the source tag name to a list of your TagStudio tag names, you can cause one source tag to import as more than one of your TagStudio tags.
By mapping the key or quoted string of a source tag to a **list of your TagStudio tags**, you can cause one source tag to import as more than one of your TagStudio tags.
```toml
[newgrounds.tags.map]
@@ -411,9 +479,11 @@ drawing = ["Drawing (2D)", "Image (Meta Tags)"]
video = ["Animation (2D)", "Animated (Meta Tags)"]
```
---
#### Many to 1 (Inverse Map)
By mapping a key with the name of one of your TagStudio tags to a list of source tags, you can declare a combination of required source tags that result in a wholly new matched TagStudio tag. This is useful if you use a single tag in your TagStudio library that is represented by multiple separate tags from your source.
By mapping the key or quoted string of one of your TagStudio tags to a **list of source tags**, you can declare a combination of required source tags that result in a wholly new matched TagStudio tag. This is useful if you use a single tag in your TagStudio library that is represented by multiple separate tags from your source.
```toml
[newgrounds.tags.inverse_map]
@@ -422,6 +492,8 @@ By mapping a key with the name of one of your TagStudio tags to a list of source
"Animation (3D)" = ["3D", "video"]
```
---
### Templates
Templates are part of the `input_data` action and allow you to take data from one or more keys of a source and combine them into a single value. Template sub-action tables must begin with the action name and end with `.template` (e.g. `[action.template]`). Source object keys can be embedded in a string value if surrounded by curly braces (`{}`). Nested keys are accessed by separating the keys with a dot (e.g. `{key.nested_key}`).

View File

@@ -464,6 +464,8 @@ def _fill_template(
if isinstance(value, dict):
for v in value:
# NOTE: This f-string is the only thing defining how the nested key syntax works.
# If instead you wanted to use key[nested] syntax for example, use: f"{key}[{str(v)}]"
normalized_key: str = f"{key}.{str(v)}"
template = _fill_template(template, value, str(v), normalized_key)