Summary

• Problems with integration testing
• Possible solutions
• Testcontainers

Definition

Test interacting with external components

• Internal microservice API
• data broker (Kafka, Kinesis)
• database (PostgreSQL, Mongo, Redis, HBase)
• object storage (S3, HDFS)

Problems encountered

Using mocks

Mock may not implement the exact system behavior

• you may not encounter problems you would encounter on real system
• you may encounter problems that do not exist on real system
• you could miss a usefull feature (painless language plugin on elasticsearch)

Local test instances

Complex setup for each developper

• PostgreSQL, MongoDB, Redis, Elasticsearch, Kafka, HBase, etc… in a single stack
• Support and document different OS/packaging systems or use docker-compose
• High resource consumption

Dedicated test instance of a service

Availability

You have to ensure the instance is available or the test will fail.

Test configuration complexity

You have to inject credentials & addresses for each service.

Mutualized instance

Conflict between tests

• Parallel execution
  • writing data at the same time can modify each other data
• Serial execution
  • remaining data of previous tests fails the current test

Need to manage data unicity

• Work on different DB or topics per test
• Clean data after test

Solution

• An instance per test/test suite to avoid unwanted interaction like mock
• An real implementation of the service like dedicated test instance

Let me introduce test containers

Library allowing to programatically launch services to use in tests in containers

• Services launched in the test/test suite
  • availablility
  • unicity
• No setup required except docker compatible environment (sic)
• Only launch services needed for a given test: resource efficient

Main implementation in Java

Existing implementations in Python, NodeJS, Rust, but they may not be on par with the Java one.

Support many popular services out of the box

Databases

• PostgreSQL,
• MongoDB,
• MySQL/MariaDB
• InfluxDB, Neo4J, Oracle, Presto, etc…

Data Brokers

• Kafka,
• RabbitMQ,
• Pulsar

AWS Services

Using localstack (compatible implementations, but at the limit of the mock)

• S3,
• Kinesis,
• DynamoDB,
• Lambda,
• SQS, SNS,
• other services

Elasticsearch

Mockserver

To mock external Rest API

Docker compose

To launch a complete environment at once

Your microservices

Only condition is that it is dockerised.

You can extend the library and reuse services accross tests

How to use it?

Examples

Mongo

val mongo = new MongoDBContainer(DockerImageName.parse("mongo:4.0.10"))
mongo.start()
val uri = s"mongodb://${mongo.getContainerIpAddress}:${mongo.getPort}/$database"

// and then you can create your client...

Elasticsearch

  val node: ElasticsearchContainer = new ElasticsearchContainer(
    DockerImageName
      .parse("docker.elastic.co/elasticsearch/elasticsearch")
      .withTag("5.6.14")
  )
  node.start()

  
  val javaClient: TransportClient = {
    val transportAddress = new InetSocketTransportAddress(node.getTcpHost())
    ...
  }
  lazy val httpClient: HttpClient = HttpClient(
    ElasticsearchClientUri("elasticsearch://" + node.getHttpHostAddress)
  )

Kafka

  val node: KafkaContainer = new KafkaContainer(
      DockerImageName.parse("confluentinc/cp-kafka:6.2.1")
    )
  node.start()
  
  val settings = ConsumerSettings(
      system, 
      new LongDeserializer, 
      new StringDeserializer
    ) // or any other deserializer
      .withBootstrapServers(kafkaContainer.getBootstrapServers)
      .withGroupId("group-id")
      .withProperty(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest")

Avoid traps

Container lifecyle

Don’t forget to start the container, don’t forget to stop it.

One container per test suite

• Reference to the container as a field of the class
• Start the container during the test initialization
• Create the clients
• Use BeforeAfterAll to override afterAll to stop the container

A container per test

Cannot have a single reference to the container like previously.

How to create a container per test and stop it each time?

Get a ready to use container, without worrying about its lifecycle

"Kafka consumer" should {
  "read all messages on a topic" in  withKafka { kafkaContainer =>

      // Test your stuff

  }
}

What about CI?

Support in every CI environment able to run Docker containers

See how

Gitlab CI

Mount the Docker socket in the container

See how

Drone

Using a plugin