Ingestion Troubleshooting
General Troubleshooting
When encountering issues with ingestion, follow these steps to diagnose and potentially resolve the problem:
Enable diagnostic logs: To enhance error tracking, configure your OTLP client to capture comprehensive diagnostic logs by following the instructions listed below.
Verify endpoint connectivity: Use a simple shell command like
curl <endpoint>from the machine in question to determine if there are any local network configuration issues (such as firewall restrictions) preventing connectivity to the ingestion endpoint. You can find your endpoint here.Validate Ingestion Key: Ensure your Ingestion key is valid and properly set. Invalid or missing Ingestion keys are a common issue that typically present with HTTP 403 or 401 status codes, or gRPC Unauthenticated or PermissionDenied status codes. If you encounter these, double-check that your Ingestion key is valid, correctly configured and has not expired.
Validate Ingestion Limits: Incorrect Ingestion key limits may inadvertently restrict your data flow. Verify your current Ingestion limits to ensure optimal data throughput. For a comprehensive understanding of how these limits function, refer to our Ingestion Key Limits documentation.
Evaluate retry success: Confirm if exports succeed after retry attempts. While occasional transient errors are expected, persistent failures may indicate a more significant issue. If you suspect that transient errors are occurring more frequently than expected, please reachout to SigNoz support.
Monitor non-retried errors: Be on the lookout for scenarios where transient errors are not being retried. Despite best efforts, there may be corner cases where the ingestion endpoint returns non-retriable status codes for transient errors. If you believe you've encountered this scenario, please reachout to SigNoz support.
Collect Diagnostic Logs
Diagnostic Logs capture critical information that helps identify root causes of issues and provides essential context for troubleshooting, whether performed by you or the SigNoz Support team.
Opentelemetry Collector
In order to enable detailed log information, add the following snippet to your collector configuration:
service:
telemetry:
logs:
level: DEBUG
Language SDKs
Refer to the SigNoz instrumentation docs for language-specific details.
Issues Catalog
Root Cause:
- Payload is malformed
Resolution:
- Depending upon the endpoint, consult the relevant documenation to verify the payload schema.
By systematically working through these steps, you can often identify and resolve common Ingestion issues. For persistent problems that you can't resolve through these methods, please reachout to SigNoz support.
