Skip to content

Publish partial step values on application shutdown for step meters #1882

Closed

Description

@timtebeek

Hi! Had posted this in Slack, but thought to report it here as well pending response there

In short: I have a Spring Boot 2.2.5 application with spring-boot-starter-actuator and micrometer-registry-elastic; when I set management.metrics.export.elastic.enabled=true, then my counters do not increment anymore.

Not sure if this should be a micrometer issue or related to Spring Boot; ElasticMeterRegistry is in micrometer whereas ElasticMetricsExportAutoConfiguration is in Spring Boot. But since the latter seems to mostly delegate and is rather minimal otherwise, I thought to report it here first.

To replicate the problem:

  1. Create a new project with: https://start.spring.io/#!type=maven-project&language=java&platformVersion=2.2.5.RELEASE&packaging=jar&jvmVersion=11&groupId=com.example&artifactId=demo&name=demo&description=Demo%20project%20for%20Spring%20Boot&packageName=com.example.demo&dependencies=actuator

  2. Add a dependency on micrometer-registry-elastic

<dependency>
	<groupId>io.micrometer</groupId>
	<artifactId>micrometer-registry-elastic</artifactId>
</dependency>
  1. Create a test class containing
@Test
void counterStaysAtZero() {
	DebugApplication.main("", "--management.metrics.export.elastic.enabled=true");
	// DebugApplication : Counter: 0.0 , [Measurement{statistic='COUNT', value=0.0}]
	// DebugApplication : Counter: 0.0 , [Measurement{statistic='COUNT', value=0.0}]
}
@Test
void counterAddsToOne() {
	DebugApplication.main("", "--management.metrics.export.elastic.enabled=false");
	// DebugApplication : Counter: 0.0 , [Measurement{statistic='COUNT', value=0.0}]
	// DebugApplication : Counter: 1.0 , [Measurement{statistic='COUNT', value=1.0}]
}
@SpringBootApplication
@RequiredArgsConstructor
@Slf4j
public static class DebugApplication implements ApplicationRunner {
	public static void main(String... args) {
		log.info("{}", args[1]);
		SpringApplication.run(DebugApplication.class, args).close();
	}
	private final MeterRegistry registry;
	@Override
	public void run(ApplicationArguments args) throws Exception {
		Counter counter = registry.counter("someCounter");
		log.info("Counter: {} , {}", counter.count(), counter.measure());
		counter.increment();
		log.info("Counter: {} , {}", counter.count(), counter.measure());
	}
}

If you then run both tests you will get the log output that's listed in each test method.

I would expect the Counter to increment regardless of whether metrics are exported; What can be done to make this so the counters increase?

Activity

  1. izeye commented on Mar 7, 2020

    @izeye
    Contributor

    @timtebeek Thanks for the issue!

    ElasticMeterRegistry is a StepMeterRegistry which uses StepCounter for Counter whereas SimpleMeterRegistry, which is used when there's no backend in Spring Boot, uses CumulativeCounter. That's the reason why you see the behavioral difference. StepCounter reports the number of events in the last complete interval as mentioned in its Javadoc, so if you check the value after its step, you can see what you expected.

    I updated your sample to demonstrate what I said here.

  2. added
    questionA user question, probably better suited for StackOverflow
    on Mar 7, 2020
  3. timtebeek commented on Mar 7, 2020

    @timtebeek
    Author

    Thanks for the quick response @izeye ! Had no idea StepMeterRegistry and StepCounter were involved to give this effect. It affects us most for short lived processes/jobs that increment a counter at the end of their run, before a last gasp push of metrics to Elastic. In these cases the behaviour we're seeing is that the counters are created with value 0, but aren't actually incremented before the final push and the process dies.

    Are you suggesting we add a step sized (~1 minute) sleep to the end of our jobs to ensure the incremented counter values are pushed?

    Or is there another way to force pushing all "current" values as a program is shutting down, rather than wait for a step duration?

  4. izeye commented on Mar 7, 2020

    @izeye
    Contributor

    @timtebeek The step-sized sleep was just for the demonstration, but I couldn't think of any way to push a value in an incomplete step with the current implementation except waiting for the completion of the step.

  5. timtebeek commented on Mar 9, 2020

    @timtebeek
    Author

    Can confirm adding the one minute sleep at the end of our jobs resolved the immediate issue we have with counters not showing up incremented, so thanks for that @izeye !

    Now you've indicated that there's presently no way to push a value in an incomplete step, even when an application is shutting down and metrics are pushed as a result of that shutdown hook. Would it be worthwhile exploring such a use-case and keep this ticket open? Or do you suggest we close this one?

  6. izeye commented on Mar 9, 2020

    @izeye
    Contributor

    @timtebeek Thanks for the confirmation!

    I think it would be nice if we could push a value in an incomplete step rather than waiting for a step to be completed in case of shutdown although I don't know how we could achieve it at the moment.

    @shakuzen Thoughts?

  7. changed the title [-]Counters do not increment when ElasticMeterRegistry export is enabled[/-] [+]StepCounter does not push incomplete step values on application shutdown[/+] on Mar 9, 2020
  8. timtebeek commented on Mar 9, 2020

    @timtebeek
    Author

    Perfect thanks; I've (hopefully) updated the title to reflect this change in scope (feel free to refine as needed)

    As a user of the micrometer for jobs it would be great if on application shutdown any values in incomplete metric steps are still pushed to Elastic or whatever else is configured.

  9. shakuzen commented on Mar 10, 2020

    @shakuzen
    Member

    I believe we have a previous ticket along these same lines - it came up for Influx as the backend if I remember correctly, but same use case. I do think we should have a better solution for this. I'll link the issues and add it to an upcoming release once I get done with the current release crunch.

  10. timtebeek commented on Mar 10, 2020

    @timtebeek
    Author

    Awesome, thanks for considering such a change!

  11. pnaw94 commented on Aug 27, 2020

    @pnaw94

    Would really appreciate this one.

    We run a big-data processing platform with a large number of processes, both long and short lived (where short-lived is even <10s, and long-lived even several hours, and we can't easily assume upfront whether a process will be short or long living), and we essentially have the same issue here - metrics from short-lived processes are not being published, therefore we really only have a subset of our processes being measured.

    Putting a thread sleep call is not ideal for us, as due to high number of short-lived processes we run, this might raise our costs significantly. I can also see that there is a possibility of reducing this step size, but then I'm concerned about potential performance impact on long-running processes.

    @shakuzen I'm wondering if you could recommend a preferred approach here. Perhaps we can modify the step at runtime? I'm not sure however if it's realistically achievable. I'd really appreciate your feedback on this.

  12. admoca60 commented on Jan 13, 2021

    @admoca60

    Dear @shakuzen

    Any news about this feature? I think that Timers are in the same situation.

  13. garridobarrera commented on Mar 11, 2021

    @garridobarrera

    Hi @shakuzen
    I have a similar use case but with Influx Meter Registry. In a graceful shutdown it don't report metrics due to incomplete step. I have really interested in send metrics with short-lived processes or restarted services use cases.
    It has any estimated date to this feature?

    Regards

  14. garridobarrera commented on Mar 20, 2021

    @garridobarrera

    Please review this PR #2519

  15. 44 remaining items

  16. ppawlowski15 commented on Mar 19, 2023

    @ppawlowski15

    OK, everything is clear now. It's just my misunderstanding about this issue.

    I've created my custom PushMeterRegistry and now its work like a charm without any workarounds.

    Thanks for Your help and time.

  17. lenin-jaganathan commented on Apr 5, 2023

    @lenin-jaganathan
    Contributor

    @shakuzen / @jonatan-ivanov Now that the issues around this fix are completed. Can this be considered a candidate for 1.9.x patch?

  18. shakuzen commented on Apr 5, 2023

    @shakuzen
    Member

    Can this be considered a candidate for 1.9.x patch?

    Eventually, I think so. Recent follow-up fixes related to this are only available in snapshots so far, and we haven't received much feedback that people have tried this and verified it is working well for them and not causing any other issues. 1.11.0-RC1 will be released on Monday. Hopefully that will be an opportunity to get more people to try it out. After that, we can consider the risk of backporting, and I'm cautiously optimistic we can do it. We haven't got to a GA release with the current solution yet, so there may be some more tinkering, and I would rather do that in milestone/RC releases than in GA patch releases.

    Edit: opened #3759 for backporting

  19. supriya-albal-polestar commented on Sep 4, 2023

    @supriya-albal-polestar

    @ppawlowski15 @jonatan-ivanov I am running into similar issue while publishing metrics from lambda.
    If I set, "cloudwatch.step" value to 1 second and if my lambda finishes before 1 second, I am not seeing any metrics.
    Shutdown hooks do not work for AWS lambda. From your discussions, I am not clear on how I can implement custom PushMeterRegistry as CloudWatchRegistry is extended from StepRegistry.
    Any examples or pointers would be helpful. Thanks!

  20. shakuzen commented on Sep 4, 2023

    @shakuzen
    Member

    @supriya-albal-polestar could you please open a new issue with details of what you're seeing? As far as I remember, shutdown hooks do work on AWS Lambda for Java, but the process will not be shutdown after each invocation, and it may take some time after the last invocation before the process is shutdown.

  21. supriya-albal-polestar commented on Sep 5, 2023

    @supriya-albal-polestar

    Thanks @shakuzen . I have raised - #4060.
    I tried waiting for some long time, still was not able to see metrics on adding shutdown hook.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions