April 5, 2025
Random News

techlinkway

Correlate telemetry knowledge with Amazon OpenSearch Service and Amazon Managed Grafana

Admin028 mins
Correlate telemetry knowledge with Amazon OpenSearch Service and Amazon Managed Grafana


Troubleshooting a big, complicated, distributed enterprise software entails challenges like tracing requests throughout a number of companies, figuring out efficiency bottlenecks throughout the stack, and understanding cascading failures between dependent companies. Prospects typically have to work with remoted knowledge to establish the underlying reason behind the issue. By correlating totally different alerts like logs, traces, metrics, and different efficiency indicators, you will get worthwhile perception into what triggered the issue, the place, and why.

Amazon OpenSearch Service is a managed service to deploy, function, and search knowledge at scale inside AWS. Amazon Managed Grafana is a safe knowledge visualization service to question operational knowledge from a number of sources, together with OpenSearch Service.

On this publish, we present you the way to use these companies to correlate the assorted observability alerts that enhance root trigger evaluation, thereby leading to diminished Imply Time to Decision (MTTR). We additionally present a reference answer that can be utilized at scale for proactive monitoring of enterprise functions to keep away from an issue earlier than they happen.

Answer overview

The next diagram reveals the answer structure for gathering and correlating varied enterprise telemetry alerts at scale.

On the core of this structure are functions composed of microservices (represented by orange containers) working on Amazon Elastic Kubernetes Service (Amazon EKS). These microservices comprise instrumentation that emit telemetry knowledge within the type of metrics, logs, and traces. This knowledge is exported into the OpenTelemetry Collector, which serves as a central vendor agnostic gateway to gather this knowledge uniformly.

On this publish, we use an OpenTelemetry demo software as a pattern enterprise software. Giant enterprise prospects sometimes separate their observability sign knowledge into varied shops for scalability, fault isolation, entry management, and ease of operation. To help in these features, we advocate and use Amazon OpenSearch Ingestion for a serverless, scalable, and absolutely managed knowledge pipeline. We separate log and hint knowledge and ship them to distinct OpenSearch Service domains. The answer additionally sends the metrics knowledge to Amazon Managed Service for Prometheus.

We use Amazon Managed Grafana as a knowledge visualization and analytics platform to question and visualize this knowledge. We additionally present the way to make use of correlations as a worthwhile software to realize insights from these alerts unfold throughout varied knowledge shops.

The next sections define constructing this structure at scale.

Stipulations

Full the next prerequisite steps:

  1. Provision and configure the Amazon Managed Prometheus workspace to obtain metrics from the OpenTelemetry Collector.
  2. Create two devoted OpenSearch Service domains (or use current ones) to ingest logs and traces from the OpenTelemetry Collector.
  3. Create an Amazon Managed Grafana workspace and configure knowledge sources to connect with Amazon Managed Prometheus and OpenSearch Service.
  4. Arrange an EKS cluster to deploy functions and the OpenTelemetry Collector.

Create log and hint OpenSearch Ingestion pipelines

Earlier than establishing the ingestion pipelines, that you must create the mandatory AWS Id and Entry Administration (IAM) insurance policies and roles. This course of entails creating two insurance policies for area and OSIS entry, adopted by making a pipeline function that makes use of these insurance policies.

Create a coverage for ingestion

Full the next steps to create an IAM coverage:

  1. Open the IAM console.
  2. Select Insurance policies within the navigation pane, then select Create coverage.
  3. On the JSON tab, enter the next coverage into the editor:
{
    "Model": "2012-10-17",
    "Assertion": [
        {
            "Effect": "Allow",
            "Action": "es:DescribeDomain",
            "Resource": "arn:aws:es:*:{accountId}:domain/*"
        },
        {
            "Effect": "Allow",
            "Action": [ "es:ESHttpGet", "es:HttpHead", "es:HttpDelete", "es:HttpPatch", "es:HttpPost", "es:HttpPut" ],
            "Useful resource": "arn:aws:es:us-east-1:{accountId}:area/otel-traces"
        },
        {
            "Impact": "Permit",
            "Motion": [ "es:ESHttpGet", "es:HttpHead", "es:HttpDelete", "es:HttpPatch", "es:HttpPost", "es:HttpPut" ],
            "Useful resource": "arn:aws:es:us-east-1:{accountId}:area/otel-logs"
        }
        }
    ]
}

// Substitute {accountId} with your individual values
  1. Select Subsequent, select Subsequent once more, and title your coverage domain-policy.
  2. Select Create coverage.
  3. Create one other coverage with the title osis-policy and use the next JSON:
{
    "Model": "2012-10-17",
    "Assertion": [
        {
            "Effect": "Allow",
            "Action": "osis:Ingest",
            "Resource": "arn:aws:osis:us-east-1:{accountId}:pipeline/osi-pipeline-otellogs"
        },
        {
            "Effect": "Allow",
            "Action": "osis:Ingest",
            "Resource": "arn:aws:osis:us-east-1:{accountId}:pipeline/osi-pipeline-oteltraces"
        }
    ]
}
// Substitute {accountId} with your individual values

Create a pipeline function

Full the next steps to create a pipeline function:

  1. On the IAM console, select Roles within the navigation pane, then select Create function.
  2. Choose Customized belief coverage and enter the next coverage into the editor:
{
    "Model": "2012-10-17",
    "Assertion": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service": [
                    "eks.amazonaws.com",
                    "osis-pipelines.amazonaws.com"
                ],
                "AWS": "{nodegroup_arn}"
            },
            "Motion": "sts:AssumeRole"
        }
    ]
}

// Substitute {nodegroup_arn} with your individual values
  1. Select Subsequent, then seek for and choose the insurance policies osis-policy and domain-policy you simply created.
  2. Select Subsequent and title the function PipelineRole.
  3. Select Create function.

Permit entry for the pipeline function in OpenSearch Service domains

To allow entry for the pipeline function in OpenSearch Service domains, full the next steps:

  1. Open the OpenSearch Service console.
  2. Select your area (both logs or traces).
  3. Select the OpenSearch Dashboards URL
  4. Check in along with your credentials.

Then, full the next steps for every OpenSearch Service area (logs and traces domains).

  1. In OpenSearch Dashboards, go to the Safety
  2. Select Roles after which all_access.

This process makes use of the all_access function for demonstration functions solely. This grants full administrative privileges to the pipeline function, which violates the precept of least privilege and will pose safety dangers. For manufacturing environments, you need to create a customized function with minimal permissions required for knowledge ingestion, restrict permissions to particular indexes and operations, contemplate implementing index patterns and time-based entry controls, and usually audit function mappings and permissions. For detailed steering on creating customized roles with applicable permissions, check with Safety in Amazon OpenSearch Service.

  1. Select Mapped customers after which Managed mapping.
  2. On the Map consumer web page, below Backend roles, replace the backend function with the Amazon Useful resource Title (ARN) for the function PiplelineRole.
  3. Select Map.

Create a pipeline for logs

Full the next steps to create a pipeline for logs:

  1. Open the OpenSearch Service console.
  2. Select Ingestion pipelines.
  3. Select Create pipeline.
  4. Outline the pipeline configuration by coming into the next:
model: "2"
otel-logs-pipeline:
  supply:
    otel_logs_source:
      path: "/v1/logs"
  sink:
    - opensearch:
        hosts: ["{OpenSearch_domain_endpoint}"]
        aws:
          sts_role_arn: "arn:aws:iam::{accountId}:function/osi-pipeline-role"
          area: "us-east-1"
          serverless: false
        index: "observability-otel-logs%{yyyy-MM-dd}"
       
 # To get the values for the placeholders:
 # 1. {OpenSearch_domain_endpoint}: You could find the area endpoint by navigating to the Amazon Managed Opensearch managed clusters within the AWS Administration Console, after which clicking on the area.
 # After acquiring the mandatory values, change the placeholders within the configuration with the precise values.           

Create a pipeline for traces

Full the next steps to create a pipeline for traces:

  1. Open the OpenSearch Service console.
  2. Select Ingestion pipelines.
  3. Select Create pipeline.
  4. Outline the pipeline configuration by coming into the next:
model: "2"
entry-pipeline:
  supply:
    otel_trace_source:
      path: "/v1/traces"
  processor:
    - trace_peer_forwarder:
  sink:
    - pipeline:
        title: "span-pipeline"
    - pipeline:
        title: "service-map-pipeline"
span-pipeline:
  supply:
    pipeline:
      title: "entry-pipeline"
  processor:
    - otel_traces:
  sink:
    - opensearch:
        index_type: "trace-analytics-raw"
        hosts: ["{OpenSearch_domain_endpoint}"]
        aws:                 
          sts_role_arn: "arn:aws:iam::{accountId}:function/osi-pipeline-role"
          area: "us-east-1"
service-map-pipeline:
  supply:
    pipeline:
      title: "entry-pipeline"
  processor:
    - service_map:
  sink:
    - opensearch:
        index_type: "trace-analytics-service-map"
        hosts: ["{OpenSearch_domain_endpoint}"]
        aws:                 
          sts_role_arn: "arn:aws:iam::{accountId}:function/osi-pipeline-role"
          area: "us-east-1"
         
 # To get the values for the placeholders:
 # 1. {OpenSearch_domain_endpoint}: You could find the area endpoint by navigating to the Amazon Managed Opensearch managed clusters within the AWS Administration Console, after which clicking on the area.  # 2. {accountId}: That is your AWS account ID. You could find your account ID by clicking in your username within the top-right nook of the AWS Administration Console and choosing "My Account" from the dropdown menu.
 # After acquiring the mandatory values, change the placeholders within the configuration with the precise values.     

Set up the OpenTelemetry demo software in Amazon EKS

Use the EKS cluster you arrange earlier together with AWS CloudShell or one other software to finish these steps:

  1. Open the AWS Administration Console.
  2. Select the CloudShell icon within the high navigation bar, or go on to the CloudShell console.
  3. Look forward to the shell setting to initialize—it comes preinstalled with frequent AWS Command Line Interface (AWS CLI) instruments.

Now you may full the next steps to put in the applying.

  1. Clone the OpenTelemetry Demo repository:
git clone https://github.com/aws-samples/sample-correlation-opensearch-repository
  1. Navigate to the Kubernetes listing:
cd deployment_files
  1. Deploy the demo software utilizing kubectl apply:
kubectl apply -f .
  1. Use a load balancer to show the frontend service so you may attain the supply software net URL:
kubectl expose deployment opentelemetry-demo-frontendproxy --type=LoadBalancer --name=frontendproxy
  1. After you’ve got deployed the applying, entry the frontend software utilizing the load balancer on port 8080. Use your browser to go to http://:8080/ to open the supply software for OpenTelemetry.

By following these steps, you may efficiently set up and entry demo functions in your EKS cluster.

Configure the OpenTelemetry Collector exporter for logs, traces, and metrics

The OpenTelemetry Collector is a software that manages the receiving, processing, and exporting of telemetry knowledge out of your software to a goal repository.

On this step, we ship logs and traces to OpenSearch Service and metrics to Amazon Managed Prometheus. The OpenTelemetry Collector additionally works with common knowledge repositories like Jaeger and quite a lot of different open supply and industrial platforms. On this part, we embody steps to configure the OpenTelemetry Collector in an EKS setting. Then we deploy the demo software and discover the OpenTelemetry exporters utilizing AWS Managed Options as an alternative of the open supply variations.

Full the next steps:

  1. Open the otel-collector-config ConfigMap in your most popular editor:
kubectl edit configmap opentelemetry-demo-otelcol -n otel-demo
  1. Replace the exporters part with the next configuration (present the suitable Amazon Managed Service for Prometheus endpoint and OpenSearch Service log ingestion URLs):
exporters:
     logging: {}
     otlphttp/logs:
       logs_endpoint: "/v1/logs"
       auth:
         authenticator: sigv4auth
       compression: none
     otlphttp/traces:
       traces_endpoint: "/v1/traces"
       auth:
         authenticator: sigv4auth
       compression: none
     prometheusremotewrite:
        endpoint: ""
        auth:
          authenticator: sigv4auth 
  1. Find the extensions part and replace the IAM function ARN within the sigv4auth configuration:
sigv4auth:
        assume_role:
            arn: "arn:aws:iam::{accountId}:function/osi-pipeline-role"
            sts_region: "us-east-1"
        area: "us-east-1"
        service: "osis"
 #  {accountId}: change accountID along with your account id
  1. After updating the ConfigMap, restart the OpenTelemetry Collector deployment:
kubectl rollout restart deployment opentelemetry-demo-otelcol -n otel-demo

With these modifications, the OpenTelemetry Collector will ship hint knowledge to the OpenSearch Service area, metrics knowledge to the AWS Managed Service for Prometheus endpoint, and log knowledge to the OpenSearch Service area.

Configure Amazon Managed Grafana

Earlier than you may visualize your logs and traces, that you must configure OpenSearch Service as a knowledge supply in your Amazon Managed Grafana workspace. This configuration is finished by way of the Amazon Managed Grafana console.

Configure the OpenSearch Service knowledge supply

Full the next steps to configure the OpenSearch Service knowledge supply:

  1. Open the Amazon Managed Grafana console.
  2. Choose your workspace and select the workspace URL to entry your Grafana occasion.
  3. Log in to your Amazon Managed Grafana occasion.
  4. From the facet menu, select the configuration (gear) icon.
  5. On the Configuration menu, select Information Sources.
  6. Select Add knowledge supply.
  7. On the Add knowledge supply web page, choose OpenSearch Service from the checklist of accessible knowledge sources.
  8. Within the Title discipline, enter a descriptive title for the info supply.
  9. Within the URL discipline, enter the URL (OpenSearch Service area endpoint) of your OpenSearch Service area, together with the protocol and port quantity.
  10. In case your OpenSearch cluster is configured with authentication, present the required credentials within the Person and Password
  11. If you wish to use a selected index sample for the info supply, you may specify it within the Index title discipline (For instance, logstash-*).
  12. Regulate another settings as wanted, such because the Time discipline title and Time interval.
  13. Select Save & Take a look at to confirm the connection to your OpenSearch cluster.

If the take a look at is profitable, you need to see a inexperienced notification with the message “Information supply is working.”

  1. Select Save to save lots of the info supply configuration.
  2. Repeat the identical steps for the OpenSearch logs and traces domains.

Configure the Prometheus knowledge supply

Full the next steps to configure the Prometheus knowledge supply:

  1. Open the Amazon Managed Grafana console.
  2. Choose your workspace and select the workspace URL to entry your Grafana occasion.
  3. Log in to your Amazon Managed Grafana occasion.
  4. From the facet menu, select the configuration (gear) icon.
  5. On the Configuration menu, select Information Sources.
  6. Select Add knowledge supply.
  7. On the Add knowledge supply web page, choose Amazon Managed Prometheus from the checklist of accessible knowledge sources.
  8. Within the Title discipline, enter a descriptive title for the info supply.
  9. The AWS Auth Supplier and Default Area fields ought to be routinely populated based mostly in your Amazon Managed Grafana workspace configuration.
  10. Within the Workspace discipline, enter the ID or alias of your Amazon Managed Prometheus workspace.
  11. Select Save & Take a look at to confirm the connection to your Amazon Managed Prometheus workspace.

If the take a look at is profitable, you need to see a inexperienced notification with the message “Information supply is working.”

  1. Select Save to save lots of the info supply configuration.

Create correlations in Amazon Managed Grafana

To determine connections between your logs and traces knowledge, that you must arrange knowledge correlations in Amazon Managed Grafana. This lets you navigate seamlessly between associated logs and traces. Observe these steps in your Amazon Managed Grafana workspace:

  1. Open the Amazon Managed Grafana console.
  2. Choose your workspace and select the workspace URL to entry your Grafana occasion.
  3. Within the Amazon Managed Grafana portal, on the Administration menu, select Plugins and Information, and select Correlation.

  1. On the Arrange the goal for the correlation web page, below Goal, select your traces knowledge supply (OpenSearch Service, for instance, otel-traces) from the dropdown checklist and outline the question that may execute when the hyperlink is adopted. You need to use variables to question particular discipline values. For instance, traceId: ${__value.uncooked}.

  1. On the Arrange the goal for the correlation web page, select the log knowledge supply from the dropdown checklist, and enter the sphere title to be linked or correlated with the traces knowledge supply within the OpenSearch Service knowledge supply. For instance, traceID.

  1. Select Save to finish the correlation configuration.

  1. Repeat the steps to create a correlation between metrics on Prometheus to logs in OpenSearch Service.

Validate outcomes

In Amazon Managed Grafana, utilizing the Prometheus knowledge supply, find the specified occasion for correlation. The occasion ID will likely be displayed as a hyperlink. Observe the hyperlink to open the corresponding log particulars in a panel on the proper facet of the web page.

With the logs to traces correlation configured, you may entry hint info straight from the logs web page. Select traces on the log particulars panel to view the corresponding hint knowledge.

The next screenshot demonstrates the node graph visualization exhibiting the correlation circulate: occasion metrics to logs to traces.

Clear up

Take away the infrastructure for this answer when not in use to keep away from incurring pointless prices.

Conclusion

On this publish, we confirmed the way to use correlation as a useful software to realize perception into observability knowledge saved in varied shops.

Separating logs and traces into devoted domains supplies the next advantages:

  • Higher useful resource allocation and scaling based mostly on totally different workload patterns
  • Unbiased efficiency optimization for every knowledge kind
  • Simplified value monitoring and administration
  • Enhanced safety management with separate entry insurance policies

You need to use this answer as a reference to construct a scalable observability answer in your enterprise to detect, examine, and remediate issues sooner. This means, when used alongside next-generation synthetic intelligence and machine studying (AI/ML), helps to not solely proactively react however predict and forestall issues earlier than they happen. You may study extra about AI/ML with AWS.

Concerning the Authors

Balaji Mohan is a Senior Supply Guide specializing in software and knowledge modernization to the cloud. His business-first strategy supplies seamless transitions, aligning know-how with organizational targets. Utilizing cloud-centered architectures, he delivers scalable, agile, and cost-effective options, driving innovation and development.

Senthil Ramasamy is a Senior Database Guide at Amazon Net Providers. He works with AWS prospects to offer steering and technical help on database companies, serving to them with database migrations to the AWS Cloud and bettering the worth of their options when utilizing AWS.

Muthu Pitchaimani is a Search Specialist with Amazon OpenSearch Service. He builds large-scale search functions and options. Muthu is within the matters of networking and safety, and is predicated out of Austin, Texas.

Leave a Reply

Your email address will not be published. Required fields are marked *

Related News