EventingTeskit : View integration tests time out

1

Kalix JVM SDKs v1.3.4 allows an easy way to write integration tests for Views using EventingTeskit :+1:

However, when I run all tests (~550), the View integration tests using EventingTestkit fail.

2023-11-29 16:27:31,178 ERROR kalix.javasdk.testkit.impl.EventingTestServiceImpl - runSource in failed
akka.http.impl.engine.HttpIdleTimeoutException: HTTP idle-timeout encountered, no bytes passed in the last 60 seconds. This is configurable by akka.http.[server|client].idle-timeout.

Running them independently, or only with a few other tests, works well.

The issue seems to be in this line EventingTestKitImpl.scala:241. This seems to be because the Http server set up has a default 60 seconds idle timeout.

Updating application.test.conf didn’t work:

akka.http {
  server.idle-timeout = 600s
  client.idle-timeout = 600s
}

Nor using the Kalix settings:

KalixTestKit
  .DefaultSettings
  .withEventSourcedEntityIncomingMessages("source")
  .withStopTimeout(600.seconds)

If I’m reading this correctly, there’s no way to configure that Http server due to this KalixTestKit:507 line.

Is the Http server idle timeout the issue? Any ideas on how I can configure it? Thank you

2

Hi!

Glad you’re finding it useful!

I’m not too sure the timeout is the issue here.

Would you be able to provide some more details? When you run all tests, do the View integration tests consistently fail or are they flaky? Are there multiple integration tests or is it multiple scenarios under the same integration test? If multiple integration tests, do they all fail?

Would you be able to run the tests with the ENV var KALIX_TESTKIT_DEBUG=true and share the logs for the failure scenario?

Thanks!

3

To simplify debugging, I have kept only one View integration test that uses EventTestKit. I have ignored all other integration tests.

class TestSpec extends AnyWordSpec with Matchers with KalixIntegrationKit {
  "TestSpec" ignore { ... }
}

So I have one View integration test along with ~400 other unit tests. The View reads two events from an EventSourcedEntity: creation and deletion. So it’s a very simple test. Note that all the other integration tests still create containers. Without setting up those containers, I’m not able to make the View integration test fail.

In this setup, the View integration test fails intermittently. Sometimes both tests fail, other times only one test fails. I have a copy of the debug lines I thought are relevant: EventTestKit debugging · GitHub

Are there multiple integration tests or is it multiple scenarios under the same integration test? If multiple integration tests, do they all fail?

I have 4 View integration tests. They all have 2 simple sub-tests. Before these 4 View integration tests, all tests (unit and integration) would pretty much always pass. After adding the 4 View integration tests, these 4 tests fail consistently when running them along with all the other tests. Sometimes, only the first of the 2 simple sub-tests would fail, other times both of the 2 simple sub-tests would fail. Let me know if more info would help.

4

Thanks for the details and the logs.

Would you be able to add the corresponding logs from the sdk side as well? I’m only interested in the logs from package kalix.javasdk.testkit if that makes it easier to filter out application-specific logs.

I will try to reproduce on our side and get back to you.

5

I have added the kalix.javasdk.testkit logs to the above linked gist. Let me know if I could provide more info.

6

Just discovered that by running sbt test all tests pass.

[info] Run completed in 2 minutes, 55 seconds.
[info] Total number of tests run: 561
[info] Suites: completed 93, aborted 0
[info] Tests: succeeded 561, failed 0, canceled 0, ignored 1, pending 0
[info] All tests passed.
[success] Total time: 195 s (03:15), completed Dec 8, 2023, 10:59:20 AM

The previous runs are with Intellij. I guess it’s either on me, or on Intellij, that consistently fail the EventTestKit related tests.

7

Ok, thanks for letting us know.

I didn’t have the chance to try to reproduce yet but I might still try. It can still be the case there is some race condition out there that only pops out in slower setups or something of that sort.

In any case, good that it behaves fine with sbt test.

1 Like