Tag Archives: salt

Transferring large files with Salt file.managed

Posted on by
Reply

Well, this took me a few hours to figure out.

If you’re going to be transferring large files using file.managed in your salt state, make sure you specify show_changes: False, otherwise salt will start trying to boil the ocean and calculate the unified diff of your enormous files.

The clue that this is the problem you're having is if one of your CPU cores pegs at 100% and your state doesn't apply in a reasonable amount of time (minutes, I guess).

Also you probably want to make sure you're not trying to use your large file as any sort of template. By default file.managed will assume no template, which is what you want for large files that aren't templates. I've never used large files which are templates, but I suspect if you tried that you'd have a bad time.

salt stack file_tree.py

Posted on by

So in my version of salt, v2017.7.4 (Nitrogen), I was getting this error when I tried to use the file_tree ext_pillar:

Failed to load ext_pillar file_tree: must be str, not bytes

So I monkey patched my version of /usr/lib/python3/dist-packages/salt/pillar/file_tree.py, changing the file from binary to text:

            contents = ''
            try:
                # 2020-05-15 jj5 - changed 'rb' to 'r', will only work with text files...
                with salt.utils.fopen(file_path, 'r') as fhr:
                    buf = fhr.read(__opts__['file_buffer_size'])
                    while buf:
                        contents += buf
                        buf = fhr.read(__opts__['file_buffer_size'])
                    if contents.endswith('\n') \
                            and _check_newline(prefix,
                                               file_name,
                                               keep_newline):
                        contents = contents[:-1]

Salt file.recurse source file not found (file encoding issue)

Posted on by

So I was running this:

/var/www/jj-web-1-www.jj5.net-sixsigma:
  file.recurse:
    - clean: True
    - user: root
    - group: root
    - dir_mode: 755
    - file_mode: 644
    - source: salt://inst/mediawiki-1.29
    - require:
      - pkg: apache2

And getting an error like this:

----------
          ID: /var/www/jj-web-1-www.jj5.net-sixsigma
    Function: file.recurse
      Result: False
     Comment: #### /var/www/jj-web-1-www.jj5.net-sixsigma/vendor/james-heinrich/getid3/getid3/module.audio.ac3.php ####
              Source file 'salt://inst/mediawiki-1.29/vendor/james-heinrich/getid3/getid3/module.audio.ac3.php?saltenv=base' not found

              #### /var/www/jj-web-1-www.jj5.net-sixsigma/vendor/james-heinrich/getid3/getid3/module.audio-video.mpeg.php ####
              Source file 'salt://inst/mediawiki-1.29/vendor/james-heinrich/getid3/getid3/module.audio-video.mpeg.php?saltenv=base' not found
     Started: 14:27:18.352264
    Duration: 134735.945 ms
     Changes:
----------

The issue was that the source files mentioned weren’t in UTF-8 format. To convert the files I ran, e.g.:

$ iconv -f WINDOWS-1252 -t UTF-8//TRANSLIT < module.audio-video.mpeg.php.bak > module.audio-video.mpeg.php

(Actually I couldn’t get the ‘iconv’ command to work so I edited manually in Vim)

Debugging JINJA in Salt Stack

Posted on by

So I was trying to figure out how to report the template source file within the template itself and I found myself wanting to know what was available in the JINJA globals for a Salt Stack JINJA template.

Turns out you use the show_full_context() function, like this:

Context is: {{ show_full_context() }}

You can parse the output to see what top-level items are available. In my case I found the ‘source’ setting for the JINJA template in use.

Hot tip: if you’re using Vim to inspect the show_full_context() output, try using ‘%’ or ‘]}’ to move between matching braces.

Bonus: while I was learning about reporting the JINJA context I discovered that you can call Salt functions from JINJA templates, like this:

# The following two function calls are equivalent.
{{ salt['cmd.run']('whoami') }}
{{ salt.cmd.run('whoami') }}