Skip to content

Cancelled status code not reported in the gRPC server metrics #5109

Closed

Description

@ceremo

Hi,

cancelled status codes are not reported in the gRPC server metrics, metrics in these cases are reported without the tag status code because GrpcServerObservationContext is not updated with the cancelled. For us it's important the gRPC server metric give us info of the status codes, and this could be improved for example modifying the onCancel method of the ObservationGrpcServerCallListener before running the stop operation, something like this:

  @Override
  public void onCancel() {
    try (final Scope scope = this.observation.openScope()) {
      super.onCancel();
    } finally {
      if (this.observation.getContext() instanceof final GrpcServerObservationContext obsCtx) {
        obsCtx.setStatusCode(Status.Code.CANCELLED);
      }
      this.observation.stop();
    }
  }

I have created a demo to reproduce easily cancelleds, the service has a delay of 5 seconds to give time to the client to cancel the request, you could make the call using postman or a simple grpc client with a deadline lower than 5 seconds.

Thanks for your time.

Activity

  1. ttddyy commented on May 17, 2024

    @ttddyy
    Contributor

    Observing the behavior of cancel, it seems the status code doesn't change even if the request is canceled.
    Instead, the ServiceCall#isCancelled returns true for canceled requests.

    So, I'm inclined to add GrpcServerObservationContext#isCancelled, which becomes a low cardinality tag, to indicate whether the request has been canceled or not, then keep the existing status code unchanged.

  2. ceremo commented on May 20, 2024

    @ceremo
    Author

    It works for me, cancellations in webflux are defined using connectionAborted.

  3. added this to the 1.14.0-M1 milestone on May 22, 2024
  4. added a commit that references this issue on May 22, 2024
    0bdca10
  5. added a commit that references this issue on Jun 11, 2024
    f4be539
  6. JordiMartinezVicent commented on Jul 11, 2024

    @JordiMartinezVicent

    Hi @marcingrzejszczak @ttddyy

    With the soluction provided, it creates a new event with status cancelled but it could be useful to add the cancel information to the GrpcServerObservationContext which is used to retrieve the timer metric.

    The same behavior is resolved in webflux adding a boolean at ServerRequestObservationContext

  7. jonatan-ivanov commented on Jul 11, 2024

    @jonatan-ivanov
    Member

    You mean you simply want this info added to the context even if it is not used here so that you can create an ObservationFilter or an ObservationHandler and use it?

    Or do you want the instrumentation also add a low cardinality key-value about the fact of the cancellation (that can also mean removing the event)?

    Either case, could you create a new issue and explain what you want and how you want to use this information?

    I think right now you can hack this: it's not great but you can create an ObservationHandler that receives the cancellation event (onEvent) method and reacts to that (e.g.: sets a field/keyvalue on the context, etc).

  8. JordiMartinezVicent commented on Jul 12, 2024

    @JordiMartinezVicent

    You mean you simply want this info added to the context even if it is not used here so that you can create an ObservationFilter or an ObservationHandler and use it?

    Or do you want the instrumentation also add a low cardinality key-value about the fact of the cancellation (that can also mean removing the event)?

    I think that both would be helpful. On the one hand, we want to differentiate that the timer metric generated has been cancelled. And on the other hand, we have a custom GrpcServerObservationConvention to add extra information, which would be helpful to know if the request has been cancelled.

    And yes, at least in our case, the event would have no sense.

    Either case, could you create a new issue and explain what you want and how you want to use this information?

    Perfect!

    I think right now you can hack this: it's not great but you can create an ObservationHandler that receives the cancellation event (onEvent) method and reacts to that (e.g.: sets a field/keyvalue on the context, etc).

    I will explore it, thanks for the suggestion!

  9. JordiMartinezVicent commented on Jul 12, 2024

    @JordiMartinezVicent

    @jonatan-ivanov I have just created the following issue #5301

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions