add kibana

2026-11-09 11:42:38 +08:00
1 parent 5b09d0d21e
commit 36f4038324
@@ -0,0 +1,344 @@
+
+# Purpose
+Create a small **fake application log environment** with realistic log entries, use Kibana's Discover interface to practise KQL (Kibana Query Language). By the end, you'll have:
+1. A test Elasticsearch index containing application logs.
+2. Different log levels: INFO, WARN, ERROR, DEBUG.
+3. Different application services.
+4. HTTP request and response logs.
+5. User login and authentication logs.
+6. Sample KQL queries to practise searching, filtering, and troubleshooting.
+
+We'll use a fictional application called `myapp`.
+
+Folow the following steps,
+
+# Step 1. Create test log data
+
+You can insert test data directly into Elasticsearch using the Kibana Dev Tools console.
+
+Open Kibana:
+
+`http://localhost:5601`
+
+Navigate to:
+
+**Management → Dev Tools → Console**
+
+Paste the following command.
+
+### Create test logs
+
+```http
+POST /_bulk
+{"index":{"_index":"logs-lab"}}
+{"@timestamp":"2026-11-09T01:00:00Z","level":"INFO","service":"auth-service","host":"server01","message":"User john successfully logged in","user":{"name":"john","id":"1001"},"source_ip":"192.168.1.10","event_type":"login","status":"success"}
+{"index":{"_index":"logs-lab"}}
+{"@timestamp":"2026-11-09T01:01:00Z","level":"INFO","service":"web-service","host":"server02","message":"HTTP request received","http":{"method":"GET","status_code":200,"url":"/api/products"},"response_time":120,"source_ip":"192.168.1.10"}
+{"index":{"_index":"logs-lab"}}
+{"@timestamp":"2026-11-09T01:02:00Z","level":"WARN","service":"payment-service","host":"server03","message":"Payment processing took longer than expected","response_time":3500,"transaction_id":"TX10001","amount":150.50,"currency":"USD"}
+{"index":{"_index":"logs-lab"}}
+{"@timestamp":"2026-11-09T01:03:00Z","level":"ERROR","service":"payment-service","host":"server03","message":"Payment transaction failed","error":{"type":"PaymentGatewayException","message":"Connection timeout"},"transaction_id":"TX10002","amount":250.00}
+{"index":{"_index":"logs-lab"}}
+{"@timestamp":"2026-11-09T01:04:00Z","level":"INFO","service":"auth-service","host":"server01","message":"User alice successfully logged in","user":{"name":"alice","id":"1002"},"source_ip":"192.168.1.20","event_type":"login","status":"success"}
+{"index":{"_index":"logs-lab"}}
+{"@timestamp":"2026-11-09T01:05:00Z","level":"ERROR","service":"auth-service","host":"server01","message":"Failed login attempt","user":{"name":"admin","id":"0"},"source_ip":"203.0.113.50","event_type":"login","status":"failed"}
+{"index":{"_index":"logs-lab"}}
+{"@timestamp":"2026-11-09T01:06:00Z","level":"DEBUG","service":"web-service","host":"server02","message":"Database connection pool initialized","connection_pool_size":20}
+{"index":{"_index":"logs-lab"}}
+{"@timestamp":"2026-11-09T01:07:00Z","level":"WARN","service":"web-service","host":"server02","message":"High response time detected","http":{"method":"GET","status_code":504,"url":"/api/orders"},"response_time":5000}
+{"index":{"_index":"logs-lab"}}
+{"@timestamp":"2026-11-09T01:08:00Z","level":"INFO","service":"inventory-service","host":"server04","message":"Product inventory updated","product_id":"P1001","quantity":150}
+{"index":{"_index":"logs-lab"}}
+{"@timestamp":"2026-11-09T01:09:00Z","level":"ERROR","service":"inventory-service","host":"server04","message":"Database connection failed","error":{"type":"DatabaseException","message":"Connection refused"}}
+```
+
+Click **Run**.
+
+You should receive a response indicating that 11 documents were indexed successfully.
+
+Note: If your Elasticsearch container requires authentication, Kibana Dev Tools normally handles the connection automatically.
+
+---
+
+# Step 2. Create a Data View in Kibana
+
+Elasticsearch stores the logs, but Kibana needs a Data View to display them in Discover.
+
+1. Open Kibana.
+2. Navigate to **Stack Management**.
+3. Select **Data Views**.
+4. Click **Create data view**.
+5. Enter:
+
+| Setting | Value |
+|---|---|
+| Name | My Application Logs |
+| Index pattern | `logs-lab*` |
+| Timestamp field | `@timestamp` |
+
+6. Click **Save data view**.
+
+Now navigate to:
+
+**Analytics → Discover**
+
+Select `My Application Logs`.
+
+You should see your 11 test log entries.
+
+**Important:** Your sample timestamps are in UTC. In Discover, change the time picker (top-right) to **Last 24 hours** if you don't immediately see the logs.
+
+# Step 3. Practise KQL searches
+
+Now comes the interesting part!
+
+In Discover, locate the KQL search bar at the top. Try these queries one by one.
+
+### Exercise 1: Search by log level
+
+Find all ERROR logs:
+
+```kql
+level: "ERROR"
+```
+
+Find all INFO logs:
+
+```kql
+level: "INFO"
+```
+
+Find all WARNING logs:
+
+```kql
+level: "WARN"
+```
+
+### Exercise 2: Search by application service
+
+Find logs generated by the payment service:
+
+```kql
+service: "payment-service"
+```
+
+Find logs generated by the authentication service:
+
+```kql
+service: "auth-service"
+```
+
+Find logs generated by either payment or inventory services:
+
+```kql
+service: ("payment-service" OR "inventory-service")
+```
+
+### Exercise 3: Combine multiple conditions
+
+Find ERROR logs generated by the payment service:
+
+```kql
+level: "ERROR" AND service: "payment-service"
+```
+
+Find all ERROR or WARN logs:
+
+```kql
+level: ("ERROR" OR "WARN")
+```
+
+Find failed login attempts:
+
+```kql
+event_type: "login" AND status: "failed"
+```
+
+Find logs from a particular IP address:
+
+```kql
+source_ip: "203.0.113.50"
+```
+
+### Exercise 4: Search using numeric conditions
+
+Find requests taking longer than 1 second:
+
+```kql
+response_time > 1000
+```
+
+Find requests returning HTTP 504:
+
+```kql
+http.status_code: 504
+```
+
+Find requests taking longer than 1 second and returning HTTP errors:
+
+```kql
+response_time > 1000 AND http.status_code >= 400
+```
+
+Find transactions with amounts greater than 200:
+
+```kql
+amount > 200
+```
+
+### Exercise 5: Search for specific text
+
+Find logs containing the word `timeout`:
+
+```kql
+message: *timeout*
+```
+
+Find logs containing `database`:
+
+```kql
+message: *database*
+```
+
+Find logs containing either `failed` or `failure`:
+
+```kql
+message: (*failed* OR *failure*)
+```
+
+### Exercise 6: Search for missing fields
+
+Find logs without a response time:
+
+```kql
+NOT response_time: *
+```
+
+Find logs containing an error field:
+
+```kql
+error.type: *
+```
+
+Find logs containing a transaction ID:
+
+```kql
+transaction_id: *
+```
+
+---
+
+# Step 4. Practise more realistic troubleshooting scenarios
+
+Now let's simulate some real-world scenarios that you might encounter when monitoring production systems.
+
+| Scenario | KQL Query |
+|---|---|
+| Find all application errors | `level: "ERROR"` |
+| Find payment failures | `service: "payment-service" AND level: "ERROR"` |
+| Find failed user authentication | `event_type: "login" AND status: "failed"` |
+| Find slow HTTP requests | `response_time >= 3000` |
+| Find HTTP server errors | `http.status_code >= 500` |
+| Find database problems | `error.type: "DatabaseException"` |
+| Find logs from server03 | `host: "server03"` |
+| Find all logs except DEBUG | `NOT level: "DEBUG"` |
+| Find payment errors or warnings | `service: "payment-service" AND level: ("ERROR" OR "WARN")` |
+
+These are useful patterns to understand because production troubleshooting often involves combining multiple conditions.
+
+---
+
+# Step 5. Practise using the Discover interface
+
+Besides writing KQL, I recommend practising these Kibana features:
+
+**1. Add fields to the table**
+
+In Discover, expand a log entry and add these fields to your displayed columns:
+
+- `@timestamp`
+- `level`
+- `service`
+- `host`
+- `message`
+- `response_time`
+
+**2. Sort logs**
+
+Click the `@timestamp` column to sort logs by time.
+
+**3. Filter using field values**
+
+Expand a document and click a field value to add it as a filter.
+
+For example, clicking `level: ERROR` will automatically create a filter.
+
+**4. Save searches**
+
+Save your KQL searches so you can reuse them later.
+
+For example:
+- All Application Errors
+- Payment Service Errors
+- Failed Login Attempts
+- Slow HTTP Requests
+
+# Step 6. Understand KQL vs Elasticsearch SQL
+
+One important thing to remember:
+
+**KQL is not SQL.**
+
+KQL is primarily a filtering language. It helps you find documents matching particular conditions.
+
+For example:
+
+```kql
+level: "ERROR" AND service: "payment-service"
+```
+
+If you want to perform SQL-like operations such as:
+- GROUP BY
+- COUNT
+- AVG
+- SUM
+- ORDER BY
+
+You should explore **ES|QL (Elasticsearch Query Language)** as well.
+
+For example, ES|QL can perform aggregations:
+
+```sql
+FROM logs-lab
+| WHERE level == "ERROR"
+| STATS error_count = COUNT(*) BY service
+| SORT error_count DESC
+```
+
+This counts errors grouped by application service.
+
+---
+
+## Advanced topics
+
+I suggest learning Elasticsearch and Kibana in this order:
+
+| Level | Topic | What to learn |
+|---|---|---|
+| 1 | Discover | Search and explore logs |
+| 2 | KQL | Filtering and searching |
+| 3 | ES\|QL | Aggregation and data analysis |
+| 4 | Visualize | Create charts and dashboards |
+| 5 | Dashboard | Build monitoring dashboards |
+| 6 | Alerts | Configure alerts for ERROR logs |
+| 7 | Ingestion | Use Logstash, Beats or Elastic Agent |
+| 8 | Advanced | Index lifecycle management and Elasticsearch architecture |
+
+**One particularly useful next exercise:** Create a dashboard showing:
+- Total log count
+- ERROR count
+- ERROR count grouped by service
+- HTTP response time
+- Failed login attempts over time
+
+This will give you a practical introduction to how Elasticsearch and Kibana are used in real production environments.