Difference between revisions of "Data"

From PostgreSQL_wiki
Jump to: navigation, search
(→‎Conceptual Communications Model)
 
(34 intermediate revisions by one other user not shown)
Line 1: Line 1:
 
Data is under development, so subject to change (quite much hopefully).<br/>
  
Data is under development, so subject to change (quite much hopefully).<br/>
 
Nodes interact with Data according to a protocol. We suggest we comply with the AMQP standard. https://www.amqp.org
  
Nodes interact with Data according to a protocol. We suggest we comply with the AMQP standard. https://www.amqp.org
  +
= The Universal Truth =
−
= Guiding principles =
  
  +
All nodes voluntarly sharing their information give Data the opportunity to construct a Universe of information. Data on its turn provides specific information of interest to specific nodes from this Universe to these specific nodes.
  +
== Guiding principles ==
 
* Precedence Analysis
  
* Precedence Analysis
 
* Desired state is assumend
  
* Desired state is assumend
Line 8: Line 10:
 
== Conceptual Communications Model ==
  
== Conceptual Communications Model ==
 
{| class="wikitable"
  
{| class="wikitable"
−
|!Node message (Req)
 +
!style="text-align:left;"|Node message (Req)
−
|Data Response (Rep)
 +
!style="text-align:left;"|Data Response (Rep)
−
|Node Action
 +
!style="text-align:left;"|Node Action
 
|-
  
|-
 
|rowspan="2"|Request View --->
  
|rowspan="2"|Request View --->
Line 16: Line 18:
 
|Process view info
  
|Process view info
 
|-
  
|-
−
|Request for Information --->
 +
|Reply with Request for Information --->
 
|Prepare Information Feed
  
|Prepare Information Feed
  +
|-
  +
|Request for Information Feed --->
  +
|Reply with hash of Information received --->
  +
|Store received Hash
  +
|-
  +
|Request Information Feed --->
  +
|Reply with hash of Information received --->
  +
|Store received updated Hash
 
|}
  
|}
  +
== Daemons ==
  +
Data uses the cf-messagd daemon for message handling. cf-messaged answers queries and accepts feeds from nodes. Data writes feeds into a message queue in order to have an audit trail. The feeds are picked up and processed by a daemon that evaluates the feeds are propagates the data into The Universal Truth.
   
 
= Feeds =
  
= Feeds =
 
Nodes send out feeds on request. This request is triggered by a Data response to an information request. The node assumes the requested information to be available from Data, but Data can decide to ask for more information before being able to supply the requested information.
  
Nodes send out feeds on request. This request is triggered by a Data response to an information request. The node assumes the requested information to be available from Data, but Data can decide to ask for more information before being able to supply the requested information.
  +
== host ==
  +
The node notifies Data about its existence with the host, domain, role feed.
  +
<pre>
  +
{
  +
"host": {
  +
"uqhost": "$(sys.uqhost)",
  +
"domain": "$(sys.domain)",
  +
"role": "$(classify.role)"
  +
}
  +
}
  +
</pre>
  +
 
= Schemas =
  
= Schemas =
  +
A Schema is the result set of a view represented in a json container.
  +
== Roles ==
  +
<pre>
  +
{ "response": "tstutl",
  +
"roles": [
  +
{ "role":
  +
{
  +
"id": "apjums",
  +
"name": "Apache Joomla User Managed Sites"
  +
},
  +
"role":
  +
{
  +
"id": "aptutl",
  +
"name": "apt-cacher-ng mirror server for debian, red hat and suse platforms"
  +
}
  +
}
  +
]
  +
}
  +
</pre>
  +
== Nagios ==
  +
<pre>
  +
{
  +
"response": "mnmutl",
  +
"domains": [{
  +
"domain": "webhuis.nl",
  +
"services": [{
  +
"service": "http",
  +
"hosts": ["apjums0001", "mnmutl0001"]
  +
}, {
  +
"service": "tomcat",
  +
"hosts": ["webjen0001", "webjen0002"]
  +
}]
  +
}, {
  +
"domain": "hnvk.nl",
  +
"services": [{
  +
"service": "http",
  +
"hosts": ["apjums0002", "mnmutl0002"]
  +
}, {
  +
"service": "tomcat",
  +
"hosts": ["webjen0003", "webjen0004"]
  +
}]
  +
}]
  +
}
  +
</pre>
   
 
= Views =
  
= Views =
  +
Data presents information from The Universal Truth through views. cf_message obtains information from Data by presenting Data with the json-container defined by the view that corresponds to its role. Data returns the information in the json container that mirrors the request.
  +
== Sample Dialogue ==
  +
cf-agents through cf_message presents Data with a request on a view, a nagios view in this case:
  +
<pre>
  +
{ "message": "view",
  +
"query":
  +
{ "view_name": "mnmutl" }
  +
}
  +
</pre>
  +
Nagios requests information on all the services offered by all the nodes in this symplified sample Universe. Data returns a json container with information, organized by domain, service and host.
  +
<pre>
  +
{
  +
"response": "mnmutl",
  +
"domains": [{
  +
"domain": "webhuis.nl",
  +
"services": [{
  +
"service": "http",
  +
"hosts": ["apjums0001", "mnmutl0001"]
  +
}, {
  +
"service": "tomcat",
  +
"hosts": ["webjen0001", "webjen0002"]
  +
}]
  +
}, {
  +
"domain": "hnvk.nl",
  +
"services": [{
  +
"service": "http",
  +
"hosts": ["apjums0002", "mnmutl0002"]
  +
}, {
  +
"service": "tomcat",
  +
"hosts": ["webjen0003", "webjen0004"]
  +
}]
  +
}]
  +
}
  +
</pre>
  +
cf-agent on the Nagios host on its turn processes the json container supplied by Data, using the nagios templates, to construct the configuration files used by the Nagios server.
  +
 
== CMDB ==
  
== CMDB ==
   
 
== Icinga ==
  
== Icinga ==
  +
We prefer to use the Icinga server, but we did not succeed in getting the Debian webfronted going. It keeps sending out Internal Server messages, referring to the logs that do not give any clue at all.
  +
 
== Nagios ==
  
== Nagios ==
  +
Nagios only needs little information about the nodes in the Universe.
−
hostgroups, fqdn, service
  
  +
* Hostname, sys.uqhost
  +
* Domain, sys.domain
  +
* Service, delivered by the host
  +
We feed back information on Services to Nagios. Each Service has a couple of parameters and an array of hosts fulfilling the service.
  +
<pre>
  +
{ "services": [ { "ssh": [ { "check_command": ""
  +
{ "hosts": [ { "host": "<fqdn>" }, { "host": "<fqdn>" } ]
  +
}
  +
}]
  +
}
  +
</pre>
  +
Hostgroups are implied by their domain and role, so each host belongs to the domain group and the groups of its role.
  +
== View Queries ==
  +
{| class="wikitable"
  +
!style="text-align:left;"|View Name
  +
!style="text-align:left;"|Sorted Result Set
  +
!style="text-align:left;"|Selection Keys
  +
!style="text-align:left;"|Comment
  +
|-
  +
|__________________________
  +
|__________________________
  +
|__________________________
  +
|__________________________
  +
|-
  +
|Nagios View
  +
|domain, uqhost, service
  +
|host, domain, role
  +
|Nagios groups and has service details
  +
|}
  +
 
<hr>
  
<hr>
−
Return to: [[Cfengine]]
 +
Return to: [[CFEngine]]

Latest revision as of 14:59, 1 June 2022

Data is under development, so subject to change (quite much hopefully).
Nodes interact with Data according to a protocol. We suggest we comply with the AMQP standard. https://www.amqp.org

The Universal Truth

All nodes voluntarly sharing their information give Data the opportunity to construct a Universe of information. Data on its turn provides specific information of interest to specific nodes from this Universe to these specific nodes.

Guiding principles

  • Precedence Analysis
  • Desired state is assumend
  • Data holds the Universal Truth

Data presents nodes on request with node taylored views on the Universal Truth, thus supplying the nodes with the the information they require. This is the normal situation in which Data provides updates on the views. Data may respond that it needs additional information. Nodes may find Data needs to be updated with information they hold.

Conceptual Communications Model

Node message (Req) Data Response (Rep) Node Action
Request View ---> Reply with view data ---> Process view info
Reply with Request for Information ---> Prepare Information Feed
Request for Information Feed ---> Reply with hash of Information received ---> Store received Hash
Request Information Feed ---> Reply with hash of Information received ---> Store received updated Hash

Daemons

Data uses the cf-messagd daemon for message handling. cf-messaged answers queries and accepts feeds from nodes. Data writes feeds into a message queue in order to have an audit trail. The feeds are picked up and processed by a daemon that evaluates the feeds are propagates the data into The Universal Truth.

Feeds

Nodes send out feeds on request. This request is triggered by a Data response to an information request. The node assumes the requested information to be available from Data, but Data can decide to ask for more information before being able to supply the requested information.

host

The node notifies Data about its existence with the host, domain, role feed. 

{
	"host": {
		"uqhost": "$(sys.uqhost)",
		"domain": "$(sys.domain)",
		"role": "$(classify.role)"
	}
}

Schemas

A Schema is the result set of a view represented in a json container.

Roles

{ "response": "tstutl",
  "roles": [
   { "role":
    {
      "id": "apjums",
      "name": "Apache Joomla User Managed Sites"
    },
     "role":
    {
      "id": "aptutl",
      "name": "apt-cacher-ng mirror server for debian, red hat and suse platforms"
    }
  }
  ]
}

Nagios

{
	"response": "mnmutl",
	"domains": [{
		"domain": "webhuis.nl",
		"services": [{
			"service": "http",
			"hosts": ["apjums0001", "mnmutl0001"]
		}, {
			"service": "tomcat",
			"hosts": ["webjen0001", "webjen0002"]
		}]
	}, {
		"domain": "hnvk.nl",
		"services": [{
			"service": "http",
			"hosts": ["apjums0002", "mnmutl0002"]
		}, {
			"service": "tomcat",
			"hosts": ["webjen0003", "webjen0004"]
		}]
	}]
}

Views

Data presents information from The Universal Truth through views. cf_message obtains information from Data by presenting Data with the json-container defined by the view that corresponds to its role. Data returns the information in the json container that mirrors the request.

Sample Dialogue

cf-agents through cf_message presents Data with a request on a view, a nagios view in this case: 

{ "message": "view",
  "query": 
  { "view_name": "mnmutl" }
}

Nagios requests information on all the services offered by all the nodes in this symplified sample Universe. Data returns a json container with information, organized by domain, service and host. 

{
	"response": "mnmutl",
	"domains": [{
		"domain": "webhuis.nl",
		"services": [{
			"service": "http",
			"hosts": ["apjums0001", "mnmutl0001"]
		}, {
			"service": "tomcat",
			"hosts": ["webjen0001", "webjen0002"]
		}]
	}, {
		"domain": "hnvk.nl",
		"services": [{
			"service": "http",
			"hosts": ["apjums0002", "mnmutl0002"]
		}, {
			"service": "tomcat",
			"hosts": ["webjen0003", "webjen0004"]
		}]
	}]
}

cf-agent on the Nagios host on its turn processes the json container supplied by Data, using the nagios templates, to construct the configuration files used by the Nagios server.

CMDB

Icinga

We prefer to use the Icinga server, but we did not succeed in getting the Debian webfronted going. It keeps sending out Internal Server messages, referring to the logs that do not give any clue at all.

Nagios

Nagios only needs little information about the nodes in the Universe.

  • Hostname, sys.uqhost
  • Domain, sys.domain
  • Service, delivered by the host

We feed back information on Services to Nagios. Each Service has a couple of parameters and an array of hosts fulfilling the service. 

{ "services": [  { "ssh": [  { "check_command": "" 
    { "hosts": [ { "host": "<fqdn>" }, { "host": "<fqdn>" } ]
    }
  }]
}

Hostgroups are implied by their domain and role, so each host belongs to the domain group and the groups of its role.

View Queries

View Name Sorted Result Set Selection Keys Comment
__________________________ __________________________ __________________________ __________________________
Nagios View domain, uqhost, service host, domain, role Nagios groups and has service details

Return to: CFEngine

Retrieved from "https://wiki.webhuis.nl/index.php?title=Data&oldid=4272"