Editing Cfengine3 Messaging (section)

= Nodes and messaging =
When adding messages as a feature to CFEngine different positions with regards to messages exist.
* Nodes carrying information they are willing to share
* Nodes willing to process the information Nodes are willing to share
== Nodes sharing information ==
Information sharing nodes need a method that enables them to send the information they are willing to share. In the analysis use cases of types of messages have to be mapped on the basic patterns. The listeners mirror the nodes mappings.
Nodes carry different kinds of information
* Common information
* Static information
* Dynamic Information
* State Information
In the current design of CFEngine the cf-agent then has to be extended with a promise type: messages:<br/><br/>
It goes withoout saying that messaging needs authentication and autorization mechanisms.
 == Messaging and Data ==
Convergence is one of the key concepts of CFEngine and The CFEngine Roadshow is designed to convergently work towards the desired state.<br/>
=== ZMQ ===
The CFEngine Roadshow is able to apply a light weight broker less messaging system, ZeroMQ, because the information from the agent will eventually arrive. So far the Data use case only applies Asynchronous Request/Response.<br/>
Other agent in other roles are free to implement their own message exchange systems.
=== Data ===
The REQ message arrives at Data in a JSON format and Data stores the message in the feeds schema in the database, for processing, using the JSONB data type. JSON stands for JavaScript Object Notation, JSONB stands for JSON Binary.<br/>
Data runs a Python3 Object Oriented application. Data responds with a tailor made response with all the external data the host needs to configure itself.<br/>
Data delivers the data in a leveled structure:
* host 
* domain
* role
* host_role
* domain_role
== Useful links ==
* GitHub:	https://github.com/Webhuis/CFEngine-Roadshow/tree/master/
* JSONB: 	https://tembo.io/docs/postgres_guides/postgres-basics/jsonb/
=== Example CFEngine message policy ===
<pre>
bundle agent cf_message {

vars:
  "role"        string => "x2gvdt";
  "host_info"     data => parsejson('
      {
         "host": { "uqhost": "$(sys.uqhost)", "domain": "$(sys.domain)", "role": "$(role)" },
      }');

  "host_store"      string => storejson(host_info);

commands:

  "/root/client-synchronous-request-response.py '${host_store}'";

reports:

  "${this.bundle} host_store: ${host_store}";

}

body common control {

bundlesequence => { cf_message };

inputs => { "/var/cfengine/inputs/lib/3.6/stdlib.cf" };

}
</pre>
Mind the single quotes around the parameter string being the json container, when ommitted the quotes around the keys and values will be lost.

=== Message Delivery ===
We use Python to do the messaging until an alternative in C is available (work in progress). 0mq is the first messaging system to consider, others may follow. This is the Python script the agent uses to send a message:
<pre>
#!/usr/bin/python

import sys
import zmq

context = zmq.Context()

#  Socket to talk to server
print("Connecting to server...")
socket = context.socket(zmq.REQ)
socket.connect("tcp://wbhs-pkg.webhuis.nl:5555")

message = ''
# sys.argv[0] is the program filename, slice it off
for element in sys.argv[1:]:
  message = message + element

print("Sending request %s ..." % message )
socket.send( message )

#  Get the reply.
response = socket.recv()
print( response )
</pre>

== Templates, Schemas or ? ==
Messages need to be structured.
* Identity
* Version
* Envelope
* Encryption
* Signature
With CFEngine we have mechanisms in place for editing templates.
=== Identified Schemas ===
The following Schemas have been identified so far:
* Host
* Domain
* Role
* Network
* Time zone
* DNS
* LDAP
Schemas are templates for Data, thus enabling a mechanism to provision data to Nodes.

=== messages: ===
The messages promise type is an abstract messaging method. It enables the agent to send messages to whatever listener is around in the Universe. The data type of a message is a container, and the stucture is:
<pre>
{
  "message_schema_id": {
    "message_id": "$(message_id)",
    "content": "$(content)",
    "version": "$(version)"
  }
}
</pre>

== Information processing Nodes ==
Information processing Nodes receive their information through message feeds and for their part may be willing to share their information through views. Data would be an information processing Node, relying on a defined set of feeds, an information processing process that builds the Universal Truth and defined autorized views that share the information stored.
=== cf-messaged ===
In this proof of concept the listener simply adds every message to a file, queue.txt as were it a queue. Nodes in fect feed the listener with information and the listener knows how to process incoming messages. Feeds are input schemas. The example uses the typical Zeromq port 5555, but why would we not use a more typical CFEngine port 5309?
<pre>
import time
import zmq

context = zmq.Context()
socket = context.socket(zmq.REP)
socket.bind("tcp://10.68.71.184:5555")

response = "message received"
while True:
    #  Wait for next request from client
    message = socket.recv()
    print("Received request: %s " % message)

    #  Do some 'work'
    time.sleep(1)

    with open("queue.txt", "a") as queue:
      queue.write(message + "\n")

    #  Send reply back to client
    socket.send( response )

</pre>
The output would be:
<pre>
root@wbhs-pkg:~# tail -F queue.txt 
{host:{uqhost:x2gvdt0001,domain:webhuis.nl,role:x2gvdt}}
{host:{uqhost:kvmutl0004,domain:webhuis.nl,role:kvmutl}}
{host:{uqhost:x2gvdt0001,domain:webhuis.nl,role:x2gvdt}}
{host:{uqhost:x2gvdt0001,domain:webhuis.nl,role:x2gvdt}}
</pre>
When a message is received the daemon places the message in a local queue for processing. The sending node receives a hard coded response "message received", but this could be any response as a result on the side of the listener. The response is useful when it triggers the node to supply more information in more messages.