Git

z/OS Port of the Git version control system

Installation and Usage

Use the zopen package manager (QuickStart Guide) to install:

zopen install git

Building from Source

1. Clone the repository:

git clone https://github.com/zopencommunity/gitport.git
cd gitport

2. Build using zopen:

zopen build -vv

See the zopen porting guide for more details.

Documentation

Encoding considerations

Git on z/OS leverages Git's .gitattributes support to enable support for various encodings, documented here. .gitattributes can be specified globally, or locally in repositories to determine the encoding of working tree files.

Working-tree-encoding

The working-tree-encoding attribute can be used to determine the working tree encoding. For example, to convert all files from Git's internal UTF-8 encoding to IBM-1047, you can specify the following working-tree-encoding in your .gitattributes file:

* text working-tree-encoding=IBM-1047

This will result in Git on z/OS tagging all files as IBM-1047 on checkout.

If you want the working-tree-encoding to apply to the host platform only, then you can use: platform-working-tree-encoding where platform is substituted with the system name.

On z/OS, platform is zos. Therefore, the .gitattributes would be:

* text zos-working-tree-encoding=IBM-1047

If no encoding is specified, the default UTF-8 encoding is used and all files are tagged as ISO8859-1.

To find out all of the supported encodings by git, run iconv -l.

When adding files, you need to make sure that the z/OS file tag matches the working-tree-encoding. Otherwise, you may encounter an error.

.gitattributes Pattern Ordering

Important: Rules at the BOTTOM of .gitattributes take precedence over rules at the TOP.

Rules at the BOTTOM of .gitattributes take precedence over rules at the TOP. Therefore, you should place generic/wildcard rules at the TOP and specific exceptions at the BOTTOM.

Correct order (wildcard first, specific last):

# General default for all files
* zos-working-tree-encoding=ibm-1047

# Specific exceptions
.gitattributes zos-working-tree-encoding=iso8859-1
.gitignore zos-working-tree-encoding=iso8859-1
*.xml zos-working-tree-encoding=utf-8

Result: .gitattributes and .gitignore will be ISO8859-1, *.xml files will be UTF-8, and all other files will be IBM-1047.

Incorrect order (specific first, wildcard last):

# Specific rules first
.gitattributes zos-working-tree-encoding=iso8859-1
.gitignore zos-working-tree-encoding=iso8859-1

# Wildcard last - OVERRIDES EVERYTHING ABOVE!
* zos-working-tree-encoding=ibm-1047

Result: ALL files (including .gitattributes and .gitignore) will be IBM-1047 because the wildcard at the bottom takes precedence.

Important Note: If you are relying on the zos-working-tree-encoding support and you are editing your git-managed files on a non-z/OS platform, make sure that the files are encoded in UTF-8 mode. This is because Git assumes such files are encoded in UTF-8 prior to conversion. See the working-tree-encoding documentation for more details. If you insist on editing your files in a different encoding, make sure to add the working-tree-encoding to the .gitattributes to reflect the codepage:

*  zos-working-tree-encoding=ibm-1047 working-tree-encoding=iso8859-1

This indicates that the file will be encoded in IBM-1047 on z/OS, but on non-z/OS platforms, it will be encoded in iso8859-1.

Encodings and z/OS File Tags (CCSIDs)

Note: Git on z/OS now aligns the file tag (CCSID) with the git working-tree-encoding by default. Previously, there was a specific handling for UTF-8 encoded files. These files were tagged as ISO8859-1 (CCSID 819) due to z/OS Open Tools' behavior under _BPXK_AUTOCVT=ON, which doesn't auto-convert files tagged with the UTF-8 tag (CCSID 1208). Consequently, the default tag for UTF-8 encoded files is now UTF-8 (or CCSID 1208).

To adjust the default tag for UTF-8, you can configure the git setting core.utf8ccsid to 819 using the following commands:

• git config --global core.utf8ccsid 819 # Global setting, 819 represents the CCSID for the UTF8 file tag
• git config core.utf8ccsid 819 # Local setting affecting the current repository

Alternatively, you can set the GIT_UTF8_CCSID environment variable:

• export GIT_UTF8_CCSID=819 # Environment variable

The environment variable takes precedence over the git config setting.

Example

Assuming you want to clone UTF-8 encoded files with the tag UTF8 or ccsid 819 as opposed to the default ccsid (1208):

git config --global core.utf8ccsid 819 # Set the UTF-8 ccsid 819 globally
git clone https://github.com/git/git
cd git
ls -lT # you will notice that all files are now tagged as 819

Encoding conversion fallback (Transliteration)

When Git on z/OS performs encoding conversion (e.g., from UTF-8 to IBM-1047), it may encounter characters that cannot be exactly represented in the target encoding. The core.iconvtranslit setting controls how Git handles these unmappable characters.

What is Transliteration?

Transliteration uses the iconv //TRANSLIT suffix to convert unmappable characters to similar-looking alternatives:

• Accented characters â�� base characters: café â�� cafe, naïve â�� naive, Müller â�� Muller
• Special symbols â�� ASCII equivalents: � â�� EUR, © â�� (C)
• Unmappable characters â�� ? (if no alternative exists)

Configuration:

Using environment variable (recommended, takes precedence):

• export GIT_ICONV_TRANSLIT=0 or false (Default): Git will stop with an error if a character cannot be converted.
• export GIT_ICONV_TRANSLIT=1 or true: Git will use iconv's transliteration feature to substitute the character with a similar-looking one, and will issue a warning.

Using git config:

• git config --global core.iconvtranslit false (Default): Strict mode - fail on conversion errors.
• git config --global core.iconvtranslit true: Lenient mode - transliterate unconvertible characters.

Precedence order:

1. GIT_ICONV_TRANSLIT environment variable (highest priority)
2. core.iconvtranslit configuration setting
3. Default (false - strict mode)

Accepted values: true/false, yes/no, on/off, or 1/0

Example use cases:

# Example 1: Enable for a single operation (temporary)
export GIT_ICONV_TRANSLIT=1
git clone https://github.com/example/repo-with-international-chars.git
unset GIT_ICONV_TRANSLIT  # Disable after use

# Example 2: Enable globally via config (permanent)
git config --global core.iconvtranslit true
git clone https://github.com/example/repo-with-special-chars.git

# Example 3: Check current setting
git config --get core.iconvtranslit
echo $GIT_ICONV_TRANSLIT

# Example 4: Test what characters will be transliterated
echo "café naïve Müller �" | iconv -f UTF-8 -t IBM-1047//TRANSLIT

When to enable transliteration:

• â�� Cloning repositories with international characters (accents, umlauts, currency symbols)
• â�� You encounter "failed to encode" errors during clone/checkout/add
• â�� Working with documentation/comments containing special characters
• â�� You prefer approximate conversion over strict failure
• â�� Content quality isn't critical (comments, documentation, examples)

When to keep it disabled (default):

• â�� You need exact character preservation (code, data files, legal documents)
• â�� You want to be notified of encoding issues immediately
• â�� You're working with data that must not be approximated
• â�� Source code identifiers or strings that must remain exact
• â�� Data integrity is critical

Common scenarios:

Scenario	Recommendation	Reason
Cloning public repos	Enable	Many contain international chars in comments/docs
Source code only	Disable	Code should be ASCII-safe
Documentation repos	Enable	Likely to have special characters
Database schemas	Disable	Must preserve exact data
Configuration files	Disable	Must be exact
User content (names, etc.)	Depends	Consider if approximation acceptable

Error messages:

# With transliteration DISABLED (default):
$ git clone repo-with-special-chars
error: failed to encode 'file.txt' from UTF-8 to IBM-1047 
       at line 5, col 12 (char: 0xc3a9)
fatal: unable to checkout working tree

# With transliteration ENABLED:
$ export GIT_ICONV_TRANSLIT=1
$ git clone repo-with-special-chars
warning: transliteration (best-effort) conversion used for 'file.txt'
         from UTF-8 to IBM-1047 at line 5, col 12 (char: 0xc3a9)
Cloning into 'repo-with-special-chars'... done.

Checking your system's transliteration support:

# Test if your system supports transliteration
echo "café" | iconv -f UTF-8 -t IBM-1047//TRANSLIT
# If it outputs without error, transliteration is supported

# See all available encodings
iconv -l

Binary files

To specify a binary encoding, you can use the binary attribute as follows:

*.png binary

This will tag all *.png files as binary.

Untagged files

Git on z/OS does not currently support adding untagged files. Files need to be tagged before they can be added.

Multiple encodings

You can specify multiple working-tree-encoding attributes, where the later attributes overrides the initial attributes in case of an overlap.

* text working-tree-encoding=IBM-1047
*.png binary

Migration considerations

If you are migrating from Rocket Software's Git, then the good news is that Git on z/OS should be compatible.

If you encounter any issues, please open an issue under https://github.com/zopencommunity/gitport/issues.

Git Performance considerations

This section provides various strategies to improve Git performance. It covers approaches that reduce the amount of data processed by Git, fine-tuning configuration parameters, and addresses specific considerations for encoding conversions in the working tree. Each section offers explanations and examples to help users optimize Git operations in large repositories, CI/CD environments, and systems with high I/O demands.

1. Data Reduction Strategies

Data reduction strategies are centered around minimizing the amount of data that Git must download, process, or store. By reducing the data footprint, you not only decrease network usage and disk I/O but also lower the CPU cycles required during operations.

Shallow Clones

• Purpose:
  Shallow clones limit the history depth that Git downloads. Instead of cloning the entire commit history of a repository, a shallow clone (--depth=n) retrieves only the latest commits (often just the most recent commit). This is particularly useful in CI/CD pipelines or automated builds where the full commit history is not needed.

• Benefits:

  • Reduced Data Transfer: Only a subset of the commit history is downloaded, which saves bandwidth.
  • Faster Cloning: Cloning operations become much quicker as less data is processed.
  • Lower CPU and Memory Usage: With fewer commits to process, the resource consumption is significantly reduced.

• Example Command:

  git clone --depth=1 <repo-url> my-repo

  This command tells Git to perform a shallow clone with a depth of 1, meaning only the latest commit is cloned. This is particularly useful in in CI/CD build pipelines where logs are not accessed.

Sparse Checkouts

• Purpose:
  Sparse checkouts allow you to restrict the working directory to a specific subset of files or directories within the repository. This is highly beneficial for large repositories where only a few directories are required for a particular task.

• Benefits:

  • Reduced Disk Usage: Only the necessary files are checked out, saving disk space.
  • Improved Performance: Fewer files mean less overhead for file system operations, leading to faster checkout and status commands.

• Example Workflow:

  1. Clone the repository normally:

     git clone <repo-url> my-repo
     cd my-repo

  2. Initialize sparse checkout mode:

     git sparse-checkout init --cone

  3. Specify the directories to be checked out:

     git sparse-checkout set src include

  This setup ensures that only the directories src and include are present in the working directory, thereby reducing unnecessary data processing.

Avoid Downloading Large Binaries

• Purpose:
  In repositories containing large binary files or blobs that are not needed for every operation, you can instruct Git to filter these out during the cloning process. This helps in managing bandwidth and disk space effectively.

• Benefits:

  • Efficient Network Usage: By not downloading large blobs, you reduce the time and data needed for cloning.
  • Lower Processing Overhead: Git spends less time handling unnecessary large objects.

• Example Command:

  git clone --filter=blob:none <repo-url>

  This command uses the --filter=blob:none option to prevent Git from downloading any large file blobs, making the clone operation leaner and faster.

---

2. Additional Strategies

Beyond data reduction, there are several additional strategies that can further enhance Git performance by optimizing internal Git processes and leveraging system resources more effectively.

Advanced Parallelization

• Purpose:
  Git can take advantage of multiple processors by parallelizing certain operations. This includes parallel checkouts and repack operations which are critical for large repositories.

• Benefits:

  • Reduced Checkout Time: Parallel workers can process multiple files concurrently.
  • Better Resource Utilization: Full utilization of available CPU cores leads to overall performance improvement.

• Example Configuration:

  git config --global checkout.workers -1         # Use all available cores
  git config --global checkout.thresholdForParallelism 1000

  These settings instruct Git to use all available CPU cores for checkout operations and to trigger parallelism when the number of files exceeds a certain threshold.

Compression and Garbage Collection

• Lower Compression Level (core.compression):

  • Purpose: Reduce CPU usage and improves fetch and clone performance by decreasing or disabling Git object compression.
  • Configuration:

    git config --global core.compression <level>  # 0 for no compression, 1-9 for levels
    git config --global core.compression 0      # Disable compression

  • Consideration: Trade-off between CPU and disk space.

• Minimize Garbage Collection (gc.auto):

  • Purpose: Prevent performance dips by disabling automatic garbage collection.
  • Configuration:

    git config --global gc.auto 0

  • Consideration: May require manual git gc periodically.

Performance-Enhancing Features

• feature.manyFiles Optimizations:

  • Purpose: Optimize for repositories with many files, improving commands like git status and git checkout.
  • Configuration:

    git config --global feature.manyFiles true

  • Sub-options: index.skipHash, index.version, core.untrackedCache.

• core.ignoreStat:

  • Purpose: Skip lstat() calls for change detection, beneficial if lstat() is slow on your system.
  • Configuration:

    git config --global core.ignoreStat true

  • Consideration: Default is false. Evaluate lstat() performance on z/OS.

Profiling and Diagnostics

• Purpose:
  Diagnostic environment variables such as GIT_TRACE and GIT_TRACE_PERFORMANCE help identify bottlenecks in Git operations. With the added logs, this can enable targeted performance tuning based on actual system behavior.

• Benefits:

  • Insight into Operations: Detailed trace logs can reveal which steps are consuming the most time.

• Usage:
  Set the environment variable before running Git commands:

  export GIT_TRACE=1
  export GIT_TRACE_PERFORMANCE=1

  This will output detailed trace information that can be analyzed to optimize performance further.

4. Working-Tree-Encoding: Performance Considerations

The working-tree-encoding or zos-working-tree-encoding attribute is designed to repository contents to a different encoding in the working directory. Although this is useful for projects that operate on a different encoding, it comes at a performance cost due to the on-the-fly conversions performed by the iconv library.

How Working-Tree-Encoding Works

When you define a working-tree-encoding in a .gitattributes file, Git automatically converts files from the repository's storage encoding to the specified encoding in the working tree during checkout. Conversely, when files are added or modified, Git converts them back to the repository's encoding.

• Conversion Process:
  This conversion is handled by the iconv library, a library that transforms the file's encoding. While this ensures that files are accessible in the desired format, it introduces additional CPU overhead.

Performance Impact

• Using a Global Wildcard:
  Applying a global wildcard (i.e., *) for the working-tree-encoding attribute means that every file in the repository will undergo this conversion. For example:

  * text zos-working-tree-encoding=ibm-1047

  Impact:
  • High CPU Usage: Every file, regardless of type, is subject to encoding conversion.
  • Slower Operations: In repositories with a large number of files, this can significantly slow down checkouts, status checks, and other file operations.

Use More Specific Patterns to Reduce Overhead

• Targeting Specific File Types:
  Instead of applying the encoding conversion universally, restrict it to only those file types that require a specific encoding. For example, you may only need to convert source files, such as .cob or .c files:

  *.cob text zos-working-tree-encoding=ibm-1047
  *.c text zos-working-tree-encoding=ibm-1047

  Benefits:
  • Reduced Conversion Load: Only a subset of files is processed by iconv, alleviating the performance penalty.
  • Focused Resource Usage: System resources are concentrated on files that actually benefit from encoding conversion, improving overall efficiency.

Known Limitations on z/OS

Parallel Checkout

Status: Disabled by default for files with working-tree-encoding attributes.

Background:
Git's parallel checkout feature can speed up operations by using multiple workers to check out files concurrently. However, on z/OS, this creates a race condition with file tagging (chtag) when files have working-tree-encoding attributes.

Technical Details:
z/OS requires files to be tagged with their encoding (CCSID) after checkout. This tagging operation happens outside the mutex-protected attribute cache, creating a potential race condition where parallel workers might apply incorrect encoding tags to files.

Current Behavior:

• Files without working-tree-encoding: Eligible for parallel checkout â��
• Files with working-tree-encoding: Forced to serial checkout (safe) â��
• Default setting: checkout.workers = 1 (parallel disabled)

Recommendation:
It is strongly recommended to keep parallel checkout disabled (default setting) on z/OS until the encoding and file tagging features are enhanced to handle concurrent operations safely.

Configuration:

# Check current setting (should be 1 or unset)
git config checkout.workers

# Explicitly disable parallel checkout (recommended)
git config --global checkout.workers 1

# DO NOT enable parallel checkout on z/OS:
# git config checkout.workers 4  # Not recommended!

Impact:
This limitation affects repositories with many files that use working-tree-encoding. For most use cases, the serial checkout performance is acceptable. If you have performance concerns, consider:

• Using sparse checkouts to reduce the number of files checked out
• Minimizing the use of working-tree-encoding where not strictly necessary
• Using shallow clones in CI/CD environments

Future Enhancement:
Enhancements to make file tagging thread-safe are being considered for future releases.

---

Troubleshooting

TBD

Contributing

Contributions are welcome! Please follow the zopen contribution guidelines.