[APIP] Add Access Logging support#2297
Open
DDH13 wants to merge 13 commits into
Open
Conversation
DDH13
requested review from
AnuGayan,
Arshardh,
CrowleyRajapakse,
HeshanSudarshana,
HiranyaKavishani,
Krishanx92,
PasanT9,
RakhithaRR,
Tharsanan1,
VirajSalaka,
ashera96,
chamilaadhi,
dushaniw,
hisanhunais,
malinthaprasan,
pubudu538,
renuka-fernando,
senthuran16,
tgtshanika,
tharikaGitHub and
tharindu1st
as code owners
Contributor
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (4)
🚧 Files skipped from review as they are similar to previous changes (1)
📝 WalkthroughWalkthroughThe PR adds collector-scoped capture configuration, migrates analytics policy defaults and validation to collector-based settings, updates analytics runtime to honor ignored paths and log publishing, and introduces stdout JSON logging with header masking and payload truncation. ChangesAnalytics collector and log pipeline
Estimated code review effort: 4 (Complex) | ~60 minutes Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 3 | ❌ 2❌ Failed checks (2 warnings)
✅ Passed checks (3 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Warning There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure. 🔧 golangci-lint (2.12.2)level=error msg="[linters_context] typechecking error: pattern ./...: directory prefix . does not contain modules listed in go.work or their selected dependencies" Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
gateway/system-policies/analytics/analytics_headers_test.go (1)
10-59: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winAdd direct policy-method coverage for header metadata.
These tests validate the helpers, but they do not exercise
OnRequestHeadersorOnResponseHeaders. A small request-phase and response-phase test that assertsAnalyticsMetadatacontainsrequest_headers/response_headerswhen the params are enabled would catch key-name or wiring regressions in the new behavior.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@gateway/system-policies/analytics/analytics_headers_test.go` around lines 10 - 59, Add direct coverage for the policy entry points, since the current tests only hit getHeaderFlags and serializeHeaders. Add small tests around OnRequestHeaders and OnResponseHeaders that enable send_request_headers/send_response_headers, then assert the resulting AnalyticsMetadata contains request_headers and response_headers with the expected serialized values. Use the existing helpers and policy type names to locate the wiring, and make sure the assertions catch key-name or metadata-plumbing regressions.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@gateway/gateway-controller/pkg/utils/system_policies.go`:
- Around line 202-208: The analytics policy defaults in system_policies.go are
missing the legacy allow_payloads compatibility path, so older configs no longer
map correctly through the effectiveDefaults/mergeParameters flow. Update the
analytics branch in the policy-building logic to translate
cfg.Analytics.AllowPayloads into the body-related defaults before
mergeParameters runs, or inject the legacy allow_payloads parameter there so it
still controls send_request_body/send_response_body alongside the new header
flags. Keep the fix scoped to the analytics policy handling in the code path
that builds effectiveDefaults for sysPol.Name.
---
Nitpick comments:
In `@gateway/system-policies/analytics/analytics_headers_test.go`:
- Around line 10-59: Add direct coverage for the policy entry points, since the
current tests only hit getHeaderFlags and serializeHeaders. Add small tests
around OnRequestHeaders and OnResponseHeaders that enable
send_request_headers/send_response_headers, then assert the resulting
AnalyticsMetadata contains request_headers and response_headers with the
expected serialized values. Use the existing helpers and policy type names to
locate the wiring, and make sure the assertions catch key-name or
metadata-plumbing regressions.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: ba9cbd48-6ce2-43f1-82db-aa5a9bec3263
📒 Files selected for processing (12)
gateway/configs/config-template.tomlgateway/gateway-controller/pkg/config/config.gogateway/gateway-controller/pkg/utils/system_policies.gogateway/gateway-controller/pkg/utils/system_policies_test.gogateway/gateway-runtime/policy-engine/internal/analytics/analytics.gogateway/gateway-runtime/policy-engine/internal/analytics/analytics_test.gogateway/gateway-runtime/policy-engine/internal/analytics/publishers/log.gogateway/gateway-runtime/policy-engine/internal/analytics/publishers/log_test.gogateway/gateway-runtime/policy-engine/internal/config/config.gogateway/system-policies/analytics/analytics.gogateway/system-policies/analytics/analytics_headers_test.gogateway/system-policies/analytics/policy-definition.yaml
Contributor
There was a problem hiding this comment.
Actionable comments posted: 6
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@gateway/configs/config.toml`:
- Around line 78-108: The policy_configurations.backendjwt_v1.signingkey inline
private key is checked in and should be removed from the shared config. Replace
the inline secret with a non-sensitive placeholder or file reference, and make
sure the real key is loaded from deployment-specific secret storage instead of
being embedded in config.toml.
- Around line 179-181: The introspection config is checking in real provider
credentials instead of placeholders. Update the relevant entries in the config
section containing uri, clientId, and clientSecret to use non-sensitive
placeholder values or external secret references, and keep the concrete
credentials out of the repository. Make sure the same config keys remain in
place so the introspection setup still works with injected secrets.
- Around line 157-176: The opaque token introspection providers use inconsistent
token pattern key names, so make both entries in
opaquetokenauth_v1.introspectionproviders use the same exact key as the existing
local_app configuration. Update the asgardeo_app provider’s token pattern field
to match tokenPattern so the matcher is recognized consistently by the config
parser.
In `@gateway/gateway-runtime/policy-engine/internal/config/config.go`:
- Around line 66-69: The runtime AnalyticsConfig schema is missing the new
analytics settings, so add send_request_headers, send_response_headers, and
max_payload_size alongside IgnoredPathPrefixes in the config struct used by
koanf. Update AnalyticsConfig and any related config mapping so
file-config/runtime-only loading preserves the same header capture and payload
truncation fields exposed by config-template.toml, config.toml, and the
controller config.
In `@gateway/system-policies/analytics/analytics.go`:
- Around line 496-498: The streaming analytics path still buffers the full
response in analyticsStreamAccKey even when max_payload_size is set, so update
the accumulation logic in analytics.go to cap memory use during stream
collection. Adjust the response-body handling in the streaming aggregation path
that feeds accumulateResponsePayload/getPayloadFlags so it keeps only a bounded
buffer (or stops appending once the configured limit is reached), while still
allowing response_payload to be truncated for emission.
- Around line 977-999: The getMaxPayloadSize helper currently returns negative
values unchanged even though its contract says non-positive values should be
treated as 0. Update getMaxPayloadSize in analytics.go so every parsed branch
(int, int64, float64, string) normalizes any value <= 0 to 0 before returning,
and keep the fallback behavior for unset or invalid inputs unchanged.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: d4d929f1-7abf-49a6-acbf-cf8af0edd27b
📒 Files selected for processing (13)
gateway/configs/config-template.tomlgateway/configs/config.tomlgateway/gateway-controller/pkg/config/config.gogateway/gateway-controller/pkg/utils/system_policies.gogateway/gateway-controller/pkg/utils/system_policies_test.gogateway/gateway-runtime/policy-engine/internal/analytics/analytics.gogateway/gateway-runtime/policy-engine/internal/analytics/analytics_test.gogateway/gateway-runtime/policy-engine/internal/analytics/publishers/log.gogateway/gateway-runtime/policy-engine/internal/analytics/publishers/log_test.gogateway/gateway-runtime/policy-engine/internal/config/config.gogateway/system-policies/analytics/analytics.gogateway/system-policies/analytics/analytics_headers_test.gogateway/system-policies/analytics/policy-definition.yaml
💤 Files with no reviewable changes (1)
- gateway/gateway-runtime/policy-engine/internal/analytics/publishers/log_test.go
🚧 Files skipped from review as they are similar to previous changes (3)
- gateway/gateway-controller/pkg/utils/system_policies.go
- gateway/system-policies/analytics/policy-definition.yaml
- gateway/gateway-controller/pkg/config/config.go
Contributor
Author
|
@coderabbitai pause |
Contributor
✅ Action performedReviews paused. |
DDH13
force-pushed
the
main.traffic-logging
branch
from
6469102 to
18ab16b
Compare
DDH13
changed the title
DDH13
force-pushed
the
main.traffic-logging
branch
from
21e9396 to
1b52ca4
Compare
DDH13
force-pushed
the
main.traffic-logging
branch
2 times, most recently
from
81ca557 to
a8e1c81
Compare
- Introduced configuration options for enabling header capture in analytics events. - Added a new Log publisher to output analytics events to stdout. - Updated analytics system policy to handle request and response headers. - Enhanced tests to validate header capture functionality. Enhance analytics configuration with max payload size and ignored path prefixes Remove pretty Refactor analytics and traffic logging configuration to use a unified collector - Migrate analytics configuration to a new collector structure, consolidating settings for analytics and traffic logging. - Update tests to reflect changes in configuration structure and ensure compatibility with the new collector. - Deprecate old analytics settings and provide migration paths for existing configurations. - Adjust access log service server initialization to utilize the new collector settings. - Ensure that both analytics and traffic logging consumers require the collector to be enabled for proper functionality. Refactor access log configuration to unify ALS settings under [collector.als] - Updated CollectorConfig and AnalyticsConfig to replace deprecated grpc_event_server and access_logs_service fields with als. - Adjusted validation and migration logic to reflect new configuration structure. - Modified error messages to reference the new [collector.als] section. - Enhanced traffic logging functionality by introducing TrafficLog and TrafficLogDirective types, allowing for per-API logging configurations. - Implemented tests to ensure correct behavior of traffic logging, including handling of excluded headers and field selection. - Updated access logger server initialization to utilize new configuration keys. Add auto activate for collector Refactor traffic logging configuration to support max payload size and ignored path prefixes Refactor latency calculations and enhance Latencies struct with new duration fields
- Updated the collector configuration to remove the explicit `enabled` flag, making the collector implicitly active whenever a consumer (analytics or traffic logging) is enabled. - Adjusted related comments and documentation to reflect the new behavior. - Modified tests to ensure they validate the collector's state based on consumer configurations rather than an explicit flag. - Removed deprecated path ignoring functionality from traffic logging configuration. - Enhanced logging to include custom properties in traffic log events.
…ed headers and caching traffic log directives
…nd improve event structure
…d tests and migration logic
… to prevent stale configurations affecting unrelated consumers
…projection handling for improved performance and clarity
… deprecated analytics configurations
- Introduced TrafficLogLatencies struct to hold microsecond timings. - Updated prepareAnalyticEvent to compute and assign traffic log latencies. - Modified tests to validate new traffic log latency calculations. - Adjusted existing Latencies struct to maintain separation from traffic log data.
DDH13
force-pushed
the
main.traffic-logging
branch
from
c63c060 to
1723518
Compare
Contributor
Author
|
@coderabbitai resume |
Contributor
✅ Action performedReviews resumed. |
1 similar comment
Contributor
✅ Action performedReviews resumed. |
…sts for consistency
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related to #2222
This pull request introduces a new shared "collector" configuration for request/response data capture, consolidating and modernizing how analytics and traffic logging gather and transport data. The collector now serves as the central pipeline, with its own configuration section and migration logic for deprecated analytics fields. The configuration files and codebase have been updated to reflect this new structure, ensuring backward compatibility and clearer separation of concerns.
Collector pipeline introduction and migration:
collectorpackage (common/collector/collector.go) with logic to determine when the collector should be active, and to migrate deprecated analytics body-capture and transport fields onto the new collector config, preserving backward compatibility.Config) and addedCollectorConfigandTrafficLoggingConfigtypes ingateway/gateway-controller/pkg/config/config.go, reflecting the new collector pipeline and separating consumer-specific toggles. [1] [2]Configuration file updates:
gateway/configs/config-template.tomlandgateway/configs/config.tomlto introduce the[collector]section, move ALS transport tuning under[collector.als], and clarify the relationship between analytics, traffic logging, and the collector. Deprecated analytics fields are mapped with warnings for backward compatibility. [1] [2] [3] [4]Deprecation and migration handling:
allow_payloads,send_request_body,send_response_body, and ALS transport settings) are now migrated onto the collector config during validation, with warnings, to ensure existing configs continue to work. [1] [2]Testing:
common/collector/collector_test.goto verify correct migration and enablement behavior.Codebase cleanup:
log/slogin config, centralizing logging in the collector package, and updated imports accordingly. [1] [2]These changes modernize and centralize data capture configuration, making it easier to manage and extend, while preserving backward compatibility for existing deployments.