cf-agent -Kf ./example.cf
     error: readjson: data error parsing JSON file '/tmp/mydata.json': No data
     error: readjson: data error parsing JSON file '/tmp/mydata.json': No data
     error: readjson: data error parsing JSON file '/tmp/mydata.json': No data
     error: readjson: data error parsing JSON file '/tmp/mydata.json': No data
     error: readjson: data error parsing JSON file '/tmp/mydata.json': No data
     error: readjson: data error parsing JSON file '/tmp/mydata.json': No data
     error: readjson: data error parsing JSON file '/tmp/mydata.json': No data
     error: readjson: data error parsing JSON file '/tmp/mydata.json': No data
     error: readjson: data error parsing JSON file '/tmp/mydata.json': No data
     error: readjson: data error parsing JSON file '/tmp/mydata.json': No data
     error: readjson: data error parsing JSON file '/tmp/mydata.json': No data
     error: readjson: data error parsing JSON file '/tmp/mydata.json': No data
     error: readjson: data error parsing JSON file '/tmp/mydata.json': No data

I have policy that puts a data file into place and reads data from it. Why do I get so many readjson() errors and how can I suppress them?

The minimum required to reproduce the error from readjson is:

1
2
3
4
5

  bundle agent main
  {
      vars:
        "myvar" data => readjson( "/tmp/mydata.json", 1M);
  }

Now, what happens if we add a promise that manages the file? 

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

  body file control { inputs => {"$(sys.libdir)/files.cf"}; }
  bundle agent main
  {
      vars:
        "myvar" data => readjson( "/tmp/mydata.json", 1M);

      files:
        "/tmp/mydata.json"
          create => "true",
          edit_line => insert_lines( '{ "Hello": "world" }' ),
          edit_defaults => empty;
  }

We can run with verbose logging to see more details.

Snips for brevity …

  cf-agent -Kvf /tmp/example.cf
   verbose:  CFEngine Core 3.11.0
   ...
   verbose: ----------------------------------------------------------------
   verbose:  Initialization preamble 
   verbose: ----------------------------------------------------------------
   ... 
   verbose: ----------------------------------------------------------------
   verbose:  Environment discovery 
   verbose: ----------------------------------------------------------------
   ...
   verbose: Verifying the syntax of the inputs...
   verbose: Checking policy with command '"/home/nickanderson/.cfagent/bin/cf-promises" -c "./example.cf"'
     error: readjson: data error parsing JSON file '/tmp/mydata.json': No data
     error: readjson: data error parsing JSON file '/tmp/mydata.json': No data
     error: readjson: data error parsing JSON file '/tmp/mydata.json': No data
     error: readjson: data error parsing JSON file '/tmp/mydata.json': No data
   verbose: Saved policy validated marker file '/home/nickanderson/.cfagent/state/cf_promises_validated'

The four above errors are emitted during syntax validation.

  verbose: ----------------------------------------------------------------
  verbose:  Loading policy 
  verbose: ----------------------------------------------------------------
  verbose: BEGIN parsing file: ./example.cf
  verbose: END   parsing file: ./example.cf
    error: readjson: data error parsing JSON file '/tmp/mydata.json': No data
    error: readjson: data error parsing JSON file '/tmp/mydata.json': No data
  verbose: BEGIN parsing file: /home/nickanderson/.cfagent/inputs/lib/files.cf
  verbose: END   parsing file: /home/nickanderson/.cfagent/inputs/lib/files.cf
  verbose: BEGIN parsing file: /home/nickanderson/.cfagent/inputs/lib/common.cf
  verbose: END   parsing file: /home/nickanderson/.cfagent/inputs/lib/common.cf
  verbose: Running full policy integrity checks

The above errors are emitted during the resolution that occurs during parsing inputs.

  verbose: ----------------------------------------------------------------
  verbose:  Preliminary variable/class-context convergence 
  verbose: ----------------------------------------------------------------
    error: readjson: data error parsing JSON file '/tmp/mydata.json': No data
    error: readjson: data error parsing JSON file '/tmp/mydata.json': No data
  verbose: string_mustache: argument 'default:main.myvar' does not resolve to a container or a list or a CFEngine array
  ...
  verbose: Skipping promise 'DEBUG $(this.bundle): $(link) will be a symlink to $(target)' because 'if'/'ifvarclass' is not defined
    error: readjson: data error parsing JSON file '/tmp/mydata.json': No data
    error: readjson: data error parsing JSON file '/tmp/mydata.json': No data

The above errors are emitted during pre-evaluation.

 verbose: Setting minimum acceptable TLS version: 1.0
 verbose: ----------------------------------------------------------------
 verbose:  Begin policy/promise evaluation 
 verbose: ----------------------------------------------------------------
 verbose: Using bundlesequence =>  {"main"}
 verbose: B: *****************************************************************
 verbose: B: BEGIN bundle main
 verbose: B: *****************************************************************
 verbose: V: .........................................................
 verbose: V: BEGIN variables (pass 1)
   error: readjson: data error parsing JSON file '/tmp/mydata.json': No data
 verbose: V:     Computing value of 'myvar'
   error: readjson: data error parsing JSON file '/tmp/mydata.json': No data
   error: readjson: data error parsing JSON file '/tmp/mydata.json': No data

The above errors are emitted during the first pass of variables during normal order (main evaluation).

After that the json data file is created.

 verbose: P: .........................................................
 verbose: P: BEGIN promise 'promise_example_cf_8' of type "files" (pass 1)
 verbose: P:    Promiser/affected object: '/tmp/mydata.json'
 verbose: P:    Part of bundle: main
 verbose: P:    Base context class: any
 verbose: P:    Stack path: /default/main/files/'/tmp/mydata.json'[1]
 verbose: Using literal pathtype for '/tmp/mydata.json'
 verbose: No mode was set, choose plain file default 0600
    info: Created file '/tmp/mydata.json', mode 0600

No further errors happen because now that the file exists it can be successfully parsed.

How can we suppress the errors?

You can guard the vars promise based on when there is a json file present, or based on the json_copy promise itself. But there are several things to consider. What is right depends on the specifics of the behavior you are looking for.

Considerations:

  • Basing data load on file presence does not ensure the data will be fresh when the data is loaded.
  • Basing data load on a copy promise being kept or repaired is a transient condition and a brittle state.
  • A copy promise being kept or repaired nor the presence of a file on disk will tell you if the data is valid.
  • It's not always worth checking all of the things. If the policy runs periodically convergence can help us avoid perseverating. Be careful of building in too much protective logic.

I tend to just base on the file presence, its the minimum necessary to suppress the errors:

1
2
3
4
5
6

  bundle agent main
  {
      vars:
        "myvar" data => readjson( "/tmp/mydata.json", 1M)
          if => fileexists( "/tmp/mydata.json" );
  }

If you guard based on the copy_from promise being kept or repaired the variable will only populate if the agent can successfully verify that the file looks the same locally and remotely. Do you want to use stale data if you cant reach the server?

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

  bundle agent main
  {
    vars:

        # This will only trigger if the copy_from promise is KEPT or REPAIRED
        # If the server is unavailable for whatever reason and the remote client
        # is unable to verify then the variable will not be populated.

        "myvar"
          data => readjson( "/tmp/mydata.json", 1M),
          depends_on => { "ensure_data_up_to_date" };

    files:
        "/tmp/mydata.json"
          copy_from => remote_dcp( "/srv/mydata.json", $(sys.policy_hub) ),
          handle => "ensure_data_up_to_date";
  }

Perhaps you only want the variable populated if the copy_from promise has been attempted (regardless of success or failure), and that there is data on the disk, and that data is valid.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

  bundle agent main
  {
    vars:

      # Only load the json data if the json is valid

      valid_json::

        "myvar"
          data => readjson( "/tmp/mydata.json", 1M);

    classes:

      # Only validate the data if we have TRIED to update the data. We don't care
      # if it was successful or not, only that we tried.

      ensure_data_up_to_date_reached::

      # Validate with some external tool (no native function avaialable to simply test json).
      # python -m json.tool 
      # jq .
      
        "valid_json"  expression => returnszero( "/usr/bin/python -m json.tool /tmp/mydata.json", noshell);

    files:

        "/tmp/mydata.json"
          copy_from => remote_dcp( "/srv/mydata.json", $(sys.policy_hub) ),
          handle => "ensure_data_up_to_date",
          classes => results( "bundle", "ensure_data_up_to_date" );
  }