enne2
/
grafana-POC
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
Browse Source

Initial commit: Grafana + Elasticsearch POC con dashboard Activity e Tasks

main
Matteo Benedetto 2 weeks ago
commit
977c60e579
15 changed files with 24451 additions and 0 deletions
Whitespace
Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL
Unified View
Diff Options
Show Stats Download Patch File Download Diff File
  1. 94
      .github/copilot-instructions.md
  2. 6
      .gitignore
  3. 197
      DOCUMENTAZIONE_GRAFANA_ES.md
  4. BIN
      DOCUMENTAZIONE_GRAFANA_ES.pdf
  5. 22513
      data/sample.geojson
  6. 51
      docker-compose.yml
  7. 23
      fix_dashboard.py
  8. 87
      generate_mock_data.py
  9. 716
      grafana/dashboards/dashboard.json
  10. 586
      grafana/dashboards/tasks_dashboard.json
  11. 11
      grafana/provisioning/dashboards/dashboard.yml
  12. 12
      grafana/provisioning/datasources/datasource.yml
  13. 10
      loader/Dockerfile
  14. 143
      loader/load_data.py
  15. 2
      loader/requirements.txt

94
.github/copilot-instructions.md
Unescape View File

@ -0,0 +1,94 @@
# Grafana & Elasticsearch Know-How
## 1. Elasticsearch Datasource Provisioning (YAML)
Quando si configura il datasource Elasticsearch via provisioning (es. `datasource.yml`), utilizzare il blocco `jsonData` per le configurazioni specifiche.
**Campi Chiave in `jsonData`:**
* **`index`**: Definisce il pattern dell'indice (es. `geodata`, `[logs-]YYYY.MM.DD`). **Nota:** Usare questo invece del campo top-level `database` che è deprecato.
* **`timeField`**: Il nome del campo timestamp (es. `timestamp` o `@timestamp`).
* **`esVersion`**: La versione del cluster Elasticsearch (es. `7.10.0`).
* **`interval`**: L'intervallo di tempo predefinito per i raggruppamenti (es. `Daily`, `Hourly`, `1m`). **Attenzione:** Non impostare a `None` o valori non validi, altrimenti Grafana potrebbe crashare all'avvio.
* **`maxConcurrentShardRequests`**: Limita le richieste concorrenti agli shard.
**Esempio `datasource.yml`:**
```yaml
apiVersion: 1
datasources:
- name: Elasticsearch
type: elasticsearch
uid: elasticsearch-uid
access: proxy
url: http://elasticsearch:9200
isDefault: true
jsonData:
index: "geodata"
timeField: "timestamp"
esVersion: "7.10.0"
interval: "Daily"
maxConcurrentShardRequests: 5
```
## 2. Grafana Geomap Panel con Elasticsearch
Per visualizzare punti GeoJSON da Elasticsearch, si utilizza l'aggregazione **Geohash Grid**.
**Configurazione Query:**
1. **Query Type:** `Metric`.
2. **Metric:** `Count` (o altra metrica).
3. **Group By:**
* Type: `Geohash Grid`
* Field: Il campo mappato come `geo_point` (es. `location`).
* Precision: Livello di dettaglio (es. `5`).
**Configurazione Pannello (JSON Model):**
* **`geohashField`**: Nella configurazione del layer, specifica quale campo contiene l'hash. Solitamente è `key` quando si usa l'aggregazione.
* **`location.mode`**: Deve essere impostato su `geohash`.
**Troubleshooting "Mappa Vuota":**
* **Precisione:** Se troppo alta (es. 12) su una mappa zoomata indietro, i punti potrebbero non apparire.
* **Mapping:** Il campo in Elasticsearch DEVE essere di tipo `geo_point`.
* **Geohash Field:** Assicurarsi che nel pannello Geomap, sotto "Layer" -> "Location", il "Geohash field" sia impostato correttamente (spesso `key`).
**Esempio JSON Dashboard (Snippet):**
```json
{
"type": "geomap",
"targets": [
{
"bucketAggs": [
{
"type": "geohash_grid",
"field": "location",
"settings": { "precision": "5" }
}
],
"timeField": "timestamp"
}
],
"options": {
"layers": [
{
"location": {
"mode": "geohash",
"geohashField": "key"
}
}
]
}
}
```
## 3. Troubleshooting & Best Practices
* **Controllare sempre i log:** Prima di terminare un task o assumere che una fix funzioni, controllare sempre i log del container (es. `docker compose logs grafana`). Errori di provisioning o crash all'avvio sono spesso visibili solo lì.
* **Riavvio Container:** Le modifiche ai file di provisioning (`datasource.yml`, `dashboard.yml`) richiedono spesso il riavvio del container Grafana (`docker compose restart grafana`) per essere applicate.
* **Validazione YAML:** Assicurarsi che i file YAML siano validi e che l'indentazione sia corretta.
* **Datasource UID:** Se si riscontrano errori "Datasource provisioning error: data source not found", provare a rimuovere il campo `uid` dal file YAML del datasource e fare riferimento al datasource tramite il suo `name` (stringa) nel JSON della dashboard, invece che tramite l'oggetto `{ type: ..., uid: ... }`.

6
.gitignore vendored
Unescape View File

@ -0,0 +1,6 @@
grafana_data/
es_data/
__pycache__/
*.pyc
.env
*.log

197
DOCUMENTAZIONE_GRAFANA_ES.md
Unescape View File

@ -0,0 +1,197 @@
# Dashboard Activity & Tasks su Elasticsearch
**Autore:** Matteo Benedetto
**Email:** matteo.benedetto@e-geos.it
**Data:** 03/12/2025
---
## 1. Struttura dei dati in Elasticsearch
### 1.1 Indice
Tutti i dati sono indicizzati in Elasticsearch nell'indice:
- **Nome indice:** `geodata`
L'indice viene creato automaticamente dal servizio `data_loader` (contenitore Docker) oppure tramite lo script di inizializzazione usato nella fase di sviluppo.
### 1.2 Mapping principale
Il mapping dell’indice `geodata` è stato definito in modo esplicito per evitare problemi con i tipi dinamici e per supportare aggregazioni e visualizzazioni in Grafana.
Campi principali:
- `location` (`geo_point`)
- Coordinate geografiche del punto (latitudine/longitudine).
- `timestamp` (`date`)
- Data/ora dell’evento.
- `status` (`keyword`)
- Stato del ticket/attività. Valori possibili: `OPEN`, `CLOSED`, `IN_PROGRESS`, `ON_HOLD`.
- `timing_status` (`keyword`)
- Stato di puntualità, definito solo quando `status = CLOSED`.
- Valori possibili: `CLOSED_ON_TIME`, `CLOSED_LATE`.
- `lead_time` (`integer`)
- Tempo di lavorazione (in minuti) per le attività chiuse, usato per le medie nella dashboard.
- `operator` (`keyword`)
- Operatore responsabile (es. `Operator A`, `Operator B`, …).
- `duration` (`integer`)
- Durata generica dell’evento (valore grezzo usato solo come informativo).
- `task_type` (`keyword`)
- Tipo di task, usato nella dashboard **Tasks**. Valori esemplificativi:
- `Create Activity`
- `Close Activity`
- `Create WF`
- `Close WF`
- `Select DTO`
- `Report Generation`
- `Report QC`
- `min_duration` (`integer`)
- Durata minima simulata per il tipo di task (in secondi).
- `max_duration` (`integer`)
- Durata massima simulata per il tipo di task (in secondi).
- `avg_duration` (`integer`)
- Durata media simulata per il tipo di task (in secondi).
- `task_duration` (`integer`)
- Durata effettiva del singolo evento (in secondi).
- `id` (`integer`)
- Identificativo numerico univoco del record nella generazione mock.
### 1.3 Generazione e caricamento dati
- **Generazione dati**: file `generate_mock_data.py`
- Genera un file GeoJSON `data/sample.geojson` con 1000 feature.
- Popola per ogni feature:
- coordinate casuali in Europa,
- campi di stato (`status`, `timing_status`, `lead_time`),
- informazioni di task (`task_type`, durate, operatore, timestamp).
- **Caricamento in Elasticsearch**: file `loader/load_data.py`
- Attende che Elasticsearch sia disponibile.
- Crea l’indice `geodata` con il mapping descritto sopra.
- Legge `sample.geojson` e per ogni feature crea un documento con:
- `location.lat` / `location.lon`
- tutti i campi in `properties` (timestamp, status, operator, ecc.).
---
## 2. Dashboard "Activity"
La prima dashboard si chiama **"Activity"** ed è salvata in `grafana/dashboards/dashboard.json`.
### 2.1 Pannelli principali
1. **Activity Status** (table)
- **Datasource:** `Elasticsearch`
- **Query:** aggregazione `terms` sul campo `status` + metrica `count`.
- **Trasformazioni:**
- `organize` per rinominare i campi in `Status`, `Count`, `Avg Lead Time`.
- `calculateField` per calcolare il totale e la colonna `Proportion` (percentuale sul totale).
- **Colori:** override sul campo `Status` con testo colorato (OPEN giallo, CLOSED blu, IN_PROGRESS azzurro, ON_HOLD viola).
- **Uso:** fornisce il riepilogo delle attività per stato con la proporzione e il tempo medio di lavorazione.
2. **Activity Distribution** (geomap heatmap)
- **Datasource:** `Elasticsearch`
- **Query:** metrica `raw_data` con `query: "*"` e `timeField: "timestamp"`.
- **Layer:** tipo `heatmap` con:
- `location.mode = "coords"`
- `latitude = "location.lat"`, `longitude = "location.lon"`
- **Uso:** visualizza la distribuzione geografica degli eventi come heatmap sull’Europa.
3. **Timing Status** (table)
- **Datasource:** `Elasticsearch`
- **Query:** aggregazione `terms` su `timing_status` + metrica `count`.
- **Colori:**
- `CLOSED_ON_TIME` con background verde.
- `CLOSED_LATE` con background arancione.
- **Uso:** mostra il numero di attività chiuse in tempo vs in ritardo.
4. **On-Time vs Late** (pie chart)
- **Datasource:** `Elasticsearch`
- **Query:** identica a `Timing Status` (terms su `timing_status`).
- **Trasformazioni:**
- trasformazione per convertire i risultati dell’aggregazione in valori per il pie chart.
- **Uso:** rappresentazione grafica (torta) della ripartizione CLOSED_ON_TIME vs CLOSED_LATE.
5. **Activity Over Time** (time series)
- **Datasource:** `Elasticsearch`
- **Query:** `date_histogram` su `timestamp` + metrica `count`.
- **Uso:** andamento temporale del numero di eventi.
---
## 3. Dashboard "Tasks"
La seconda dashboard si chiama **"Tasks"** ed è salvata in `grafana/dashboards/tasks_dashboard.json`.
### 3.1 Tasks (tabella in alto a sinistra)
- **Datasource:** `Elasticsearch`
- **Query:**
- `terms` su `task_type` (raggruppamento per tipo di task).
- Metriche:
- `count` (Total)
- `min` su `task_duration` (Min Duration)
- `max` su `task_duration` (Max Duration)
- **Trasformazioni:**
- `organize` per rinominare i campi:
- `task_type → Task`
- `Count → Total`
- `Min task_duration → Min Duration`
- `Max task_duration → Max Duration`
- **Colori:** testo del campo `Task` colorato in azzurro.
- **Uso:** riepilogo per tipo di task con numero totale e range di durata.
### 3.2 Tasks by Operator (grafico a barre orizzontali)
- **Datasource:** `Elasticsearch`
- **Query:**
- Primo livello: `terms` su `operator` (asse Y del grafico).
- Secondo livello: `terms` su `task_type` (serie colorate).
- Metrica: `count`.
- **Trasformazioni:**
- `groupingToMatrix` (o equivalente) per trasformare l’output dell’aggregazione in una matrice con:
- `rowField = operator`
- `columnField = task_type`
- `valueField = Count`
- **Opzioni grafiche:**
- `orientation = horizontal`.
- `stacking = normal` per avere barre impilate per operatore.
- Legenda a destra con un colore per ogni tipo di task (`Create Activity`, `Close Activity`, `Create WF`, `Close WF`, `Select DTO`, `Report Generation`, `Report QC`).
- **Uso:** mostra per ogni operatore il numero di task, suddivisi per tipologia come barre orizzontali impilate.
### 3.3 Tasks (tabella in basso)
- **Datasource:** `Elasticsearch`
- **Query:**
- `terms` su `task_type` e `operator` (raggruppamento per Task + Operator).
- Metriche:
- `count` (Total)
- `min`/`max`/`avg` su `task_duration`.
- **Trasformazioni:**
- `organize` per ordinare le colonne e rinominare i campi in:
- `Task`, `Operator`, `Total`, `Min`, `Max`, `Average`.
- **Unità di misura:**
- Durate espresse in secondi, visualizzate in Grafana come secondi/minuti (in base al formato scelto nella colonna).
- **Uso:** tabella dettagliata per tipo di task e operatore, con statistiche di durata.
---
## 4. Note operative
- **Provisioning Grafana**
- I file delle dashboard sono montati nel container Grafana tramite Docker Compose.
- Ogni modifica ai file JSON (`dashboard.json`, `tasks_dashboard.json`) richiede un **riavvio del container Grafana** (`docker compose restart grafana`).
- **Datasource Elasticsearch**
- È definito tramite provisioning (file `datasource.yml`) usando:
- `jsonData.index = "geodata"`
- `jsonData.timeField = "timestamp"`
- Non viene usata la `uid` del datasource nelle dashboard: i pannelli si riferiscono al datasource tramite `"datasource": "Elasticsearch"` per evitare problemi di provisioning.
- **Rigenerazione dati**
- Per rigenerare i dati mock:
1. Eseguire `python3 generate_mock_data.py` nella root del progetto.
2. Cancellare l’indice `geodata` in Elasticsearch.
3. Riavviare il container `data_loader` in modo che ricrei l’indice con il mapping corretto e ricarichi `sample.geojson`.
Questo documento descrive lo stato corrente della soluzione (stack Docker Elasticsearch + Grafana + loader Python) usata per visualizzare dati geospaziali e metriche di attività/task nelle dashboard **Activity** e **Tasks**.

BIN
DOCUMENTAZIONE_GRAFANA_ES.pdf
View File

Binary file not shown.

22513
data/sample.geojson
View File

File diff suppressed because it is too large Load Diff

51
docker-compose.yml
Unescape View File

@ -0,0 +1,51 @@
services:
elasticsearch:
image: docker.elastic.co/elasticsearch/elasticsearch:7.17.10
container_name: elasticsearch
environment:
- node.name=elasticsearch
- discovery.type=single-node
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
- xpack.security.enabled=false
ulimits:
memlock:
soft: -1
hard: -1
volumes:
- ./es_data:/usr/share/elasticsearch/data
ports:
- "9200:9200"
networks:
- grafana-net
grafana:
image: grafana/grafana:latest
container_name: grafana
volumes:
- ./grafana_data:/var/lib/grafana
- ./grafana/provisioning:/etc/grafana/provisioning
- ./grafana/dashboards:/var/lib/grafana/dashboards
ports:
- "3000:3000"
environment:
- GF_SECURITY_ADMIN_USER=admin
- GF_SECURITY_ADMIN_PASSWORD=admin
depends_on:
- elasticsearch
networks:
- grafana-net
data_loader:
build: ./loader
container_name: data_loader
volumes:
- ./data:/app/data
depends_on:
- elasticsearch
networks:
- grafana-net
networks:
grafana-net:
driver: bridge

23
fix_dashboard.py
Unescape View File

@ -0,0 +1,23 @@
import json
file_path = 'grafana/dashboards/dashboard.json'
with open(file_path, 'r') as f:
data = json.load(f)
def fix_datasource(obj):
if isinstance(obj, dict):
if 'datasource' in obj and isinstance(obj['datasource'], dict):
if obj['datasource'].get('type') == 'elasticsearch':
obj['datasource'] = 'Elasticsearch'
for key, value in obj.items():
fix_datasource(value)
elif isinstance(obj, list):
for item in obj:
fix_datasource(item)
fix_datasource(data)
with open(file_path, 'w') as f:
json.dump(data, f, indent=2)

87
generate_mock_data.py
Unescape View File

@ -0,0 +1,87 @@
import json
import random
from datetime import datetime, timedelta
def generate_random_point():
# Coordinate approssimative dell'Europa
lon = random.uniform(-10.0, 30.0)
lat = random.uniform(35.0, 60.0)
return [lon, lat]
def generate_data(num_points=500):
features = []
operators = ["Operator A", "Operator B", "Operator C", "Operator D", "Operator E"]
# Status principali (senza timing)
statuses = ["OPEN", "CLOSED", "IN_PROGRESS", "ON_HOLD"]
# Timing status solo per CLOSED
timing_statuses = ["CLOSED_ON_TIME", "CLOSED_LATE"]
# Task types
task_types = ["Create Activity", "Close Activity", "Create WF", "Close WF", "Select DTO", "Report Generation", "Report QC"]
base_time = datetime.now()
for i in range(num_points):
status = random.choice(statuses)
operator = random.choice(operators)
duration = random.randint(10, 300)
task_type = random.choice(task_types)
# Durations per task (in secondi) - min, max, avg variano per task
min_duration = random.randint(10, 25)
max_duration = random.randint(40, 60)
avg_duration = random.randint(min_duration + 5, max_duration - 5)
task_duration = random.randint(min_duration, max_duration + 20)
# Lead time in minuti (per Avg Lead Time) - solo per CLOSED
lead_time = None
timing_status = None
if status == "CLOSED":
lead_time = random.randint(5, 120) # minuti
# 70% on time, 30% late (come in figura ~24 vs 11)
timing_status = random.choices(timing_statuses, weights=[70, 30])[0]
# Random timestamp negli ultimi 7 giorni
time_offset = random.randint(0, 7 * 24 * 60 * 60)
timestamp = (base_time - timedelta(seconds=time_offset)).isoformat()
properties = {
"status": status,
"operator": operator,
"duration": duration,
"timestamp": timestamp,
"id": i,
"task_type": task_type,
"min_duration": min_duration,
"max_duration": max_duration,
"avg_duration": avg_duration,
"task_duration": task_duration
}
# Aggiungi lead_time e timing_status solo se CLOSED
if lead_time is not None:
properties["lead_time"] = lead_time
if timing_status is not None:
properties["timing_status"] = timing_status
feature = {
"type": "Feature",
"properties": properties,
"geometry": {
"type": "Point",
"coordinates": generate_random_point()
}
}
features.append(feature)
geojson = {
"type": "FeatureCollection",
"features": features
}
with open('data/sample.geojson', 'w') as f:
json.dump(geojson, f, indent=2)
print(f"Generated {num_points} data points in data/sample.geojson")
if __name__ == "__main__":
generate_data(1000)

716
grafana/dashboards/dashboard.json
Unescape View File

@ -0,0 +1,716 @@
{
"annotations": {
"list": [
{
"builtIn": 1,
"datasource": {
"type": "grafana",
"uid": "-- Grafana --"
},
"enable": true,
"hide": true,
"iconColor": "rgba(0, 211, 255, 1)",
"name": "Annotations & Alerts",
"type": "dashboard"
}
]
},
"editable": true,
"fiscalYearStartMonth": 0,
"graphTooltip": 0,
"id": 1,
"links": [],
"panels": [
{
"datasource": "Elasticsearch",
"fieldConfig": {
"defaults": {
"color": {
"mode": "thresholds"
},
"custom": {
"align": "auto",
"cellOptions": {
"type": "auto"
},
"inspect": false
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": 0
}
]
}
},
"overrides": [
{
"matcher": {
"id": "byName",
"options": "Status"
},
"properties": [
{
"id": "mappings",
"value": [
{
"options": {
"CLOSED": {
"color": "blue",
"index": 1,
"text": "CLOSED"
},
"IN_PROGRESS": {
"color": "light-blue",
"index": 2,
"text": "IN_PROGRESS"
},
"ON_HOLD": {
"color": "purple",
"index": 3,
"text": "ON_HOLD"
},
"OPEN": {
"color": "yellow",
"index": 0,
"text": "OPEN"
}
},
"type": "value"
}
]
},
{
"id": "custom.cellOptions",
"value": {
"type": "color-text"
}
}
]
},
{
"matcher": {
"id": "byName",
"options": "Proportion"
},
"properties": [
{
"id": "unit",
"value": "percentunit"
},
{
"id": "decimals",
"value": 1
}
]
},
{
"matcher": {
"id": "byName",
"options": "Avg Lead Time"
},
"properties": [
{
"id": "unit",
"value": "m"
},
{
"id": "decimals",
"value": 0
}
]
}
]
},
"gridPos": {
"h": 8,
"w": 8,
"x": 0,
"y": 0
},
"id": 1,
"options": {
"cellHeight": "sm",
"footer": {
"countRows": false,
"fields": "",
"reducer": [
"sum"
],
"show": false
},
"showHeader": true
},
"pluginVersion": "12.2.0-16711121739",
"targets": [
{
"bucketAggs": [
{
"field": "status",
"id": "2",
"settings": {
"min_doc_count": "1",
"order": "desc",
"orderBy": "_count",
"size": "10"
},
"type": "terms"
}
],
"datasource": "Elasticsearch",
"metrics": [
{
"id": "1",
"type": "count"
},
{
"field": "lead_time",
"id": "3",
"type": "avg"
}
],
"query": "",
"refId": "A",
"timeField": "timestamp"
}
],
"title": "Activity Status",
"transformations": [
{
"id": "organize",
"options": {
"excludeByName": {},
"indexByName": {},
"renameByName": {
"Average lead_time": "Avg Lead Time",
"Count": "Count",
"status": "Status"
}
}
},
{
"id": "calculateField",
"options": {
"alias": "Total",
"mode": "reduceRow",
"reduce": {
"reducer": "sum"
},
"replaceFields": false
}
},
{
"id": "calculateField",
"options": {
"alias": "Proportion",
"binary": {
"left": "Count",
"operator": "/",
"right": "Total"
},
"mode": "binary",
"replaceFields": false
}
},
{
"id": "organize",
"options": {
"excludeByName": {
"Total": true
},
"indexByName": {
"Avg Lead Time": 3,
"Count": 1,
"Proportion": 2,
"Status": 0
},
"renameByName": {}
}
}
],
"type": "table"
},
{
"datasource": "Elasticsearch",
"fieldConfig": {
"defaults": {
"custom": {
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": 0
},
{
"color": "red",
"value": 80
}
]
}
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 16,
"x": 8,
"y": 0
},
"id": 2,
"options": {
"basemap": {
"config": {},
"name": "Layer 0",
"type": "default"
},
"controls": {
"mouseWheelZoom": true,
"showAttribution": true,
"showDebug": false,
"showMeasure": false,
"showScale": false,
"showZoom": true
},
"layers": [
{
"config": {
"blur": 4,
"radius": 5,
"weight": {
"fixed": 1
}
},
"location": {
"latitude": "location.lat",
"longitude": "location.lon",
"mode": "coords"
},
"name": "Heatmap",
"tooltip": true,
"type": "heatmap"
}
],
"tooltip": {
"mode": "details"
},
"view": {
"allLayers": true,
"id": "europe",
"lat": 48,
"lon": 15,
"noRepeat": false,
"zoom": 4
}
},
"pluginVersion": "12.2.0-16711121739",
"targets": [
{
"alias": "",
"bucketAggs": [],
"datasource": "Elasticsearch",
"metrics": [
{
"id": "1",
"settings": {
"size": "500"
},
"type": "raw_data"
}
],
"query": "*",
"refId": "A",
"timeField": "timestamp"
}
],
"title": "Activity Distribution",
"type": "geomap"
},
{
"datasource": "Elasticsearch",
"fieldConfig": {
"defaults": {
"color": {
"mode": "thresholds"
},
"custom": {
"align": "auto",
"cellOptions": {
"type": "auto"
},
"inspect": false
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": 0
}
]
}
},
"overrides": [
{
"matcher": {
"id": "byName",
"options": "Timing Status"
},
"properties": [
{
"id": "mappings",
"value": [
{
"options": {
"CLOSED_LATE": {
"color": "orange",
"index": 1,
"text": "CLOSED_LATE"
},
"CLOSED_ON_TIME": {
"color": "green",
"index": 0,
"text": "CLOSED_ON_TIME"
}
},
"type": "value"
}
]
},
{
"id": "custom.cellOptions",
"value": {
"type": "color-background"
}
}
]
}
]
},
"gridPos": {
"h": 7,
"w": 11,
"x": 0,
"y": 8
},
"id": 3,
"options": {
"cellHeight": "sm",
"footer": {
"countRows": false,
"fields": "",
"reducer": [
"sum"
],
"show": false
},
"showHeader": true
},
"pluginVersion": "12.2.0-16711121739",
"targets": [
{
"bucketAggs": [
{
"field": "timing_status",
"id": "2",
"settings": {
"min_doc_count": "1",
"order": "desc",
"orderBy": "_count",
"size": "10"
},
"type": "terms"
}
],
"datasource": "Elasticsearch",
"metrics": [
{
"id": "1",
"type": "count"
}
],
"query": "timing_status:*",
"refId": "A",
"timeField": "timestamp"
}
],
"title": "Timing Status",
"transformations": [
{
"id": "organize",
"options": {
"excludeByName": {},
"indexByName": {},
"renameByName": {
"Count": "Count",
"timing_status": "Timing Status"
}
}
}
],
"type": "table"
},
{
"datasource": "Elasticsearch",
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
}
},
"mappings": []
},
"overrides": [
{
"matcher": {
"id": "byName",
"options": "CLOSED_ON_TIME"
},
"properties": [
{
"id": "color",
"value": {
"fixedColor": "green",
"mode": "fixed"
}
}
]
},
{
"matcher": {
"id": "byName",
"options": "CLOSED_LATE"
},
"properties": [
{
"id": "color",
"value": {
"fixedColor": "red",
"mode": "fixed"
}
}
]
}
]
},
"gridPos": {
"h": 7,
"w": 13,
"x": 11,
"y": 8
},
"id": 5,
"options": {
"displayLabels": [],
"legend": {
"displayMode": "table",
"placement": "right",
"showLegend": true,
"values": [
"value"
]
},
"pieType": "pie",
"reduceOptions": {
"calcs": [
"lastNotNull"
],
"fields": "",
"values": false
},
"tooltip": {
"hideZeros": false,
"mode": "single",
"sort": "none"
}
},
"pluginVersion": "12.2.0-16711121739",
"targets": [
{
"bucketAggs": [
{
"field": "timing_status",
"id": "2",
"settings": {
"min_doc_count": "1",
"order": "desc",
"orderBy": "_count",
"size": "10"
},
"type": "terms"
}
],
"datasource": "Elasticsearch",
"metrics": [
{
"id": "1",
"type": "count"
}
],
"query": "timing_status:*",
"refId": "A",
"timeField": "timestamp"
}
],
"title": "On-Time vs Late",
"transformations": [
{
"id": "rowsToFields",
"options": {
"mappings": [
{
"fieldName": "timing_status",
"handlerKey": "field.name"
},
{
"fieldName": "Count",
"handlerKey": "field.value"
}
]
}
}
],
"type": "piechart"
},
{
"datasource": "Elasticsearch",
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisBorderShow": false,
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"barWidthFactor": 0.6,
"drawStyle": "line",
"fillOpacity": 0,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"insertNulls": false,
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "auto",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": 0
},
{
"color": "red",
"value": 80
}
]
}
},
"overrides": []
},
"gridPos": {
"h": 6,
"w": 24,
"x": 0,
"y": 15
},
"id": 4,
"options": {
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "bottom",
"showLegend": true
},
"tooltip": {
"hideZeros": false,
"mode": "single",
"sort": "none"
}
},
"pluginVersion": "12.2.0-16711121739",
"targets": [
{
"bucketAggs": [
{
"field": "timestamp",
"id": "2",
"settings": {
"interval": "auto",
"min_doc_count": "0",
"trimEdges": "0"
},
"type": "date_histogram"
}
],
"datasource": "Elasticsearch",
"metrics": [
{
"id": "1",
"type": "count"
}
],
"query": "",
"refId": "A",
"timeField": "timestamp"
}
],
"title": "Activity Over Time",
"type": "timeseries"
}
],
"preload": false,
"refresh": "",
"schemaVersion": 41,
"tags": [],
"templating": {
"list": []
},
"time": {
"from": "now-90d",
"to": "now"
},
"timepicker": {},
"timezone": "",
"title": "Activity",
"uid": "geo-dashboard-01",
"version": 17
}

586
grafana/dashboards/tasks_dashboard.json
Unescape View File

@ -0,0 +1,586 @@
{
"annotations": {
"list": []
},
"editable": true,
"fiscalYearStartMonth": 0,
"graphTooltip": 0,
"id": null,
"links": [],
"liveNow": false,
"panels": [
{
"datasource": "Elasticsearch",
"fieldConfig": {
"defaults": {
"color": {
"mode": "thresholds"
},
"custom": {
"align": "auto",
"cellOptions": {
"type": "auto"
},
"inspect": false
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
}
]
}
},
"overrides": [
{
"matcher": {
"id": "byName",
"options": "Task"
},
"properties": [
{
"id": "custom.cellOptions",
"value": {
"type": "color-text"
}
},
{
"id": "color",
"value": {
"fixedColor": "light-blue",
"mode": "fixed"
}
}
]
},
{
"matcher": {
"id": "byName",
"options": "Min Duration"
},
"properties": [
{
"id": "unit",
"value": "s"
}
]
},
{
"matcher": {
"id": "byName",
"options": "Max Duration"
},
"properties": [
{
"id": "unit",
"value": "s"
}
]
}
]
},
"gridPos": {
"h": 10,
"w": 12,
"x": 0,
"y": 0
},
"id": 1,
"options": {
"footer": {
"fields": "",
"reducer": ["sum"],
"show": false
},
"showHeader": true
},
"pluginVersion": "10.0.0",
"targets": [
{
"bucketAggs": [
{
"field": "task_type",
"id": "2",
"settings": {
"min_doc_count": "1",
"order": "desc",
"orderBy": "_count",
"size": "10"
},
"type": "terms"
}
],
"datasource": "Elasticsearch",
"metrics": [
{
"id": "1",
"type": "count"
},
{
"id": "4",
"type": "min",
"field": "task_duration"
},
{
"id": "5",
"type": "max",
"field": "task_duration"
}
],
"query": "",
"refId": "A",
"timeField": "timestamp"
}
],
"title": "Tasks",
"transformations": [
{
"id": "organize",
"options": {
"excludeByName": {},
"indexByName": {},
"renameByName": {
"task_type": "Task",
"Count": "Total",
"Min task_duration": "Min Duration",
"Max task_duration": "Max Duration"
}
}
}
],
"type": "table"
},
{
"datasource": "Elasticsearch",
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisBorderShow": false,
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "",
"axisPlacement": "auto",
"fillOpacity": 80,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"lineWidth": 1,
"scaleDistribution": {
"type": "linear"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
}
]
}
},
"overrides": [
{
"matcher": {
"id": "byName",
"options": "Create Activity"
},
"properties": [
{
"id": "color",
"value": {
"fixedColor": "blue",
"mode": "fixed"
}
}
]
},
{
"matcher": {
"id": "byName",
"options": "Close Activity"
},
"properties": [
{
"id": "color",
"value": {
"fixedColor": "green",
"mode": "fixed"
}
}
]
},
{
"matcher": {
"id": "byName",
"options": "Create WF"
},
"properties": [
{
"id": "color",
"value": {
"fixedColor": "light-blue",
"mode": "fixed"
}
}
]
},
{
"matcher": {
"id": "byName",
"options": "Close WF"
},
"properties": [
{
"id": "color",
"value": {
"fixedColor": "dark-green",
"mode": "fixed"
}
}
]
},
{
"matcher": {
"id": "byName",
"options": "Select DTO"
},
"properties": [
{
"id": "color",
"value": {
"fixedColor": "yellow",
"mode": "fixed"
}
}
]
},
{
"matcher": {
"id": "byName",
"options": "Report Generation"
},
"properties": [
{
"id": "color",
"value": {
"fixedColor": "orange",
"mode": "fixed"
}
}
]
},
{
"matcher": {
"id": "byName",
"options": "Report QC"
},
"properties": [
{
"id": "color",
"value": {
"fixedColor": "purple",
"mode": "fixed"
}
}
]
}
]
},
"gridPos": {
"h": 10,
"w": 12,
"x": 12,
"y": 0
},
"id": 2,
"options": {
"barRadius": 0,
"barWidth": 0.8,
"fullHighlight": false,
"groupWidth": 0.7,
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "right",
"showLegend": true
},
"orientation": "horizontal",
"showValue": "never",
"stacking": "normal",
"tooltip": {
"mode": "single",
"sort": "none"
},
"xTickLabelRotation": 0,
"xTickLabelSpacing": 0
},
"targets": [
{
"bucketAggs": [
{
"field": "operator",
"id": "2",
"settings": {
"min_doc_count": "1",
"order": "asc",
"orderBy": "_term",
"size": "10"
},
"type": "terms"
},
{
"field": "task_type",
"id": "3",
"settings": {
"min_doc_count": "1",
"order": "desc",
"orderBy": "_count",
"size": "10"
},
"type": "terms"
}
],
"datasource": "Elasticsearch",
"metrics": [
{
"id": "1",
"type": "count"
}
],
"query": "",
"refId": "A",
"timeField": "timestamp"
}
],
"title": "Tasks by Operator",
"transformations": [
{
"id": "groupingToMatrix",
"options": {
"columnField": "task_type",
"rowField": "operator",
"valueField": "Count",
"emptyValue": "null"
}
}
],
"type": "barchart"
},
{
"datasource": "Elasticsearch",
"fieldConfig": {
"defaults": {
"color": {
"mode": "thresholds"
},
"custom": {
"align": "auto",
"cellOptions": {
"type": "auto"
},
"inspect": false
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
}
]
}
},
"overrides": [
{
"matcher": {
"id": "byName",
"options": "Task"
},
"properties": [
{
"id": "custom.cellOptions",
"value": {
"type": "color-text"
}
},
{
"id": "color",
"value": {
"fixedColor": "light-blue",
"mode": "fixed"
}
}
]
},
{
"matcher": {
"id": "byName",
"options": "Min"
},
"properties": [
{
"id": "unit",
"value": "s"
}
]
},
{
"matcher": {
"id": "byName",
"options": "Max"
},
"properties": [
{
"id": "unit",
"value": "s"
}
]
},
{
"matcher": {
"id": "byName",
"options": "Average"
},
"properties": [
{
"id": "unit",
"value": "s"
},
{
"id": "decimals",
"value": 1
}
]
}
]
},
"gridPos": {
"h": 10,
"w": 24,
"x": 0,
"y": 10
},
"id": 3,
"options": {
"footer": {
"fields": "",
"reducer": ["sum"],
"show": false
},
"showHeader": true
},
"pluginVersion": "10.0.0",
"targets": [
{
"bucketAggs": [
{
"field": "task_type",
"id": "2",
"settings": {
"min_doc_count": "1",
"order": "desc",
"orderBy": "_count",
"size": "10"
},
"type": "terms"
},
{
"field": "operator",
"id": "3",
"settings": {
"min_doc_count": "1",
"order": "asc",
"orderBy": "_term",
"size": "10"
},
"type": "terms"
}
],
"datasource": "Elasticsearch",
"metrics": [
{
"id": "1",
"type": "count"
},
{
"id": "4",
"type": "min",
"field": "task_duration"
},
{
"id": "5",
"type": "max",
"field": "task_duration"
},
{
"id": "6",
"type": "avg",
"field": "task_duration"
}
],
"query": "",
"refId": "A",
"timeField": "timestamp"
}
],
"title": "Tasks",
"transformations": [
{
"id": "organize",
"options": {
"excludeByName": {},
"indexByName": {
"task_type": 0,
"operator": 1,
"Count": 2,
"Min task_duration": 3,
"Max task_duration": 4,
"Average task_duration": 5
},
"renameByName": {
"task_type": "Task",
"operator": "Operator",
"Count": "Total",
"Min task_duration": "Min",
"Max task_duration": "Max",
"Average task_duration": "Average"
}
}
}
],
"type": "table"
}
],
"refresh": "",
"schemaVersion": 38,
"style": "dark",
"tags": [],
"templating": {
"list": []
},
"time": {
"from": "now-90d",
"to": "now"
},
"timepicker": {},
"timezone": "",
"title": "Tasks",
"uid": "tasks-dashboard-01",
"version": 1,
"weekStart": ""
}

11
grafana/provisioning/dashboards/dashboard.yml
Unescape View File

@ -0,0 +1,11 @@
apiVersion: 1
providers:
- name: 'Default'
orgId: 1
folder: ''
type: file
disableDeletion: false
updateIntervalSeconds: 10
options:
path: /var/lib/grafana/dashboards

12
grafana/provisioning/datasources/datasource.yml
Unescape View File

@ -0,0 +1,12 @@
apiVersion: 1
datasources:
- name: Elasticsearch
type: elasticsearch
access: proxy
url: http://elasticsearch:9200
isDefault: true
jsonData:
index: "geodata"
timeField: "timestamp"
esVersion: "7.10.0"

10
loader/Dockerfile
Unescape View File

@ -0,0 +1,10 @@
FROM python:3.9-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY load_data.py .
CMD ["python", "load_data.py"]

143
loader/load_data.py
Unescape View File

@ -0,0 +1,143 @@
import json
import os
import time
import logging
from datetime import datetime
from elasticsearch import Elasticsearch, helpers
# Configurazione logging
logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')
# Configurazione Elasticsearch
ES_HOST = os.environ.get('ES_HOST', 'elasticsearch')
ES_PORT = os.environ.get('ES_PORT', '9200')
INDEX_NAME = 'geodata'
GEOJSON_FILE = '/app/data/sample.geojson'
def wait_for_elasticsearch(es):
"""Attende che Elasticsearch sia pronto."""
while True:
try:
if es.ping():
logging.info("Elasticsearch è pronto!")
break
except Exception:
pass
logging.info("In attesa di Elasticsearch...")
time.sleep(5)
def create_index(es):
"""Crea l'indice con il mapping corretto per i dati geospaziali."""
mapping = {
"mappings": {
"properties": {
"location": {
"type": "geo_point"
},
"timestamp": {
"type": "date"
},
"status": {
"type": "keyword"
},
"operator": {
"type": "keyword"
},
"duration": {
"type": "integer"
},
"timing_status": {
"type": "keyword"
},
"lead_time": {
"type": "integer"
},
"task_type": {
"type": "keyword"
},
"min_duration": {
"type": "integer"
},
"max_duration": {
"type": "integer"
},
"avg_duration": {
"type": "integer"
},
"task_duration": {
"type": "integer"
},
"id": {
"type": "integer"
}
}
}
}
if es.indices.exists(index=INDEX_NAME):
logging.info(f"L'indice '{INDEX_NAME}' esiste già.")
else:
es.indices.create(index=INDEX_NAME, body=mapping)
logging.info(f"Indice '{INDEX_NAME}' creato con successo.")
def process_geojson(file_path):
"""Legge il file GeoJSON e prepara i documenti per Elasticsearch."""
with open(file_path, 'r') as f:
data = json.load(f)
actions = []
for feature in data.get('features', []):
geometry = feature.get('geometry')
properties = feature.get('properties', {})
# Assicuriamoci che ci sia una geometria di tipo Point
if geometry and geometry.get('type') == 'Point':
lon, lat = geometry.get('coordinates')
# Creiamo il documento
doc = {
"_index": INDEX_NAME,
"_source": {
"location": {
"lat": lat,
"lon": lon
},
# Usa il timestamp presente o quello attuale
"timestamp": properties.get('timestamp', datetime.now().isoformat()),
"status": properties.get('status', 'UNKNOWN'),
"operator": properties.get('operator', 'Unknown Operator'),
"duration": properties.get('duration', 0),
**properties # Includi tutte le altre proprietà
}
}
actions.append(doc)
return actions
def main():
es = Elasticsearch([f"http://{ES_HOST}:{ES_PORT}"])
wait_for_elasticsearch(es)
create_index(es)
# Controllo se ci sono già dati per evitare duplicati
try:
count = es.count(index=INDEX_NAME)['count']
if count > 0:
logging.info(f"L'indice '{INDEX_NAME}' contiene già {count} documenti. Salto il caricamento.")
return
except Exception:
pass
if os.path.exists(GEOJSON_FILE):
logging.info(f"Caricamento dati da {GEOJSON_FILE}...")
actions = process_geojson(GEOJSON_FILE)
if actions:
helpers.bulk(es, actions)
logging.info(f"Caricati {len(actions)} documenti in Elasticsearch.")
else:
logging.warning("Nessun dato valido trovato nel file GeoJSON.")
else:
logging.error(f"File {GEOJSON_FILE} non trovato. Assicurati di montare il volume correttamente.")
if __name__ == "__main__":
main()

2
loader/requirements.txt
Unescape View File

@ -0,0 +1,2 @@
elasticsearch==7.17.0
requests
Write Preview
Loading…
Cancel
Save