...
Setting up ELK on Ubuntu:
filebeat.inputs:
- type: log
enabled: true
paths:
- "E:/AppServer/wildfly-30.0.0.Final/wildfly-30.0.0.Final/Logs/*.json"
json.keys_under_root: true
json.overwrite_keys: true
processors:
- add_fields:
target: ''
fields:
environment: hyd-sandbox
output.elasticsearch:
hosts: ["https://<private-ip>:9200"]
api_key: "<api-key>"
ssl:
certificate_authorities: ["C:/Program Files/Filebeat/http_ca.crt"]
index: "filebeat-%{[service.name]}-%{+yyyy.MM.dd}"
setup.template.name: "filebeat"
setup.template.pattern: "filebeat-*"
setup.template.settings:
index.lifecycle.name: "90day-log-policy"
index.lifecycle.rollover_alias: "filebeat"
- Install and configure ELK on UBUNTU as single node cluster.
- Filebeat Setup on Application Nodes
...
Edit Filebeat.yml file:
Check Elasticsearch:
- Test Elasticsearch by accessing its URL in a browser: curl -X GET "localhost:9200
...
- "
- If Elasticsearch is running correctly, you should see a JSON response with Elasticsearch cluster details.
Check Kibana:
- Go to Kibana’s dashboard by accessing its URL in a browser http://localhost:5601
- Navigate to the "Discover" section and check if logs are being indexed in Elasticsearch.
Traces:
APM agent
...
Copy the apm_agent folder to the server (preferably in the wildfly installation location)
...
apm_agent folder contains two files elastic-apm-agent.jar, elasticapm.properties
Add below lines in
...
elasticapm.proerties
server_url=http://<private-ip>:8200
enable_log_correlation=true
environment=hyd-sandbox
- Filebeat Setup on Application Nodes
- Filebeat is used to forward logs from our application nodes to Logstash or Elasticsearch.
- Download and install
- Download http_ca.crt to the server (preferably in the filebeat installation location)
- Create a Logs folder inside Wildfly
Do the changes in Filebeat.yml as mentioned here
Switch to Wildfly/bin/wildfly/ and edit standalone.conf.bat add below line under wildfly folder
set "JAVA_OPTS=%JAVA_OPTS% -javaagent:E:\\AppServer\\wildfly-30.0.0.Final\\wildfly-30.0.0.Final\\apm_agent\\elastic-apm-agent-1.52.0.jar"
...
check if traces are visible in APM section of kibana
kibana
restart filebeat
check for logs under filebeat in kibana
Changes need to be done in GitHub in Do the changes in github repo under ci-properties & application properties for properties for each service line.
Example:
Goto common-ci.properties
ADD a new line
logging.file.name=@env.HWC_API_LOGGING_FILE_NAME@
Goto Goto application.properties
Add below two lines:
...
4.1. Define Environment Variables in Jenkins Pipeline:
In the Jenkins pipeline, add an set the environment variable
env.HWC_API_LOGGING_FILE_NAME='E:/AppServer/wildfly-30.0.0.Final/wildfly-30.0.0.Final/Logs/hwc-api.log'
- Verifying the Setup
5.1. Check Elasticsearch:
- Test Elasticsearch by accessing its URL in a browser:
curl -X GET "localhost:9200/"
...
5.24. Check Logstash InputApplication Logs:
- Test the connection between Filebeat and Logstash:
curl -XGET 'http://your_logstash_host:5044/_cat/health'
- Verify that Filebeat is sending logs to Logstash.
Check if the logs are being generated at the specified location (e.g., logs/inventory-api.log) and whether they are being processed by Filebeat and appear in Kibana.
5.3. Check Kibana:
- Go to Kibana’s dashboard by accessing its URL in a browser: browser http://localhost:5601
- Navigate to the "Discover" section and check if logs are being indexed in Elasticsearch.
5.4. Check Application Logs:
...
- .
...
Troubleshooting
- Filebeat not sending logs: Check the Filebeat logs at /var/log/filebeat/ for errors.
- Elasticsearch or Kibana not working: Review the logs in /var/log/elasticsearch/ and /var/log/kibana/.
- Application not writing logs: Ensure that the application has access to the specified log path and that the environment variable is being correctly passed from Jenkins.
...
This document outlines the process of configuring ELK for log collection from our application. we have set up Elasticsearch, Logstash, and Kibana on Ubuntu, configured Filebeat for log forwarding, and adjusted your our application’s properties file to use environment variables for log paths. The Jenkins pipeline is configured to pass the necessary environment variables to the application, ensuring that the logs are written to the correct location.
...