Host legacy application in Docker 2 of 2

My previous notes include some tricks in hosting legacy application in docker. This is a continuation from that work, after 1.5 months…

Use Case

I decided to use docker to host application for a good reason, and let me start with what this Java-based application does as a single process. When it is up it listens to more than 70 TCP ports for different business services. Here is a simplified list:

The application also communicates with database and search engine on the same server. Since I am building a training environment where multiple instances of our application needs to run on a single server host. All these instances of application share the same underlying database and search engine services. With multiple instances, additional constraints are introduced. For example:

1. Each instance requires more than 120 configuration files. A small number of them defines what ports the process binds to. The rest of configuration files are the same across all instances.
2. The OS needs to host 6 processes of the same application all running at the same time;
3. The OS does not allow multiple processes to bind to a single TCP port (duh!);
4. It is extremely labourious to change the path for application to read configuration files from. This bad configuration also breaks the upgrade process going forward.

From the statements of constraints, I determine that we need a mechanism to bring running application process into an isolated environment. This is exactly the definition of container and a perfect use case for docker. The following table represents an example of how the multiple instances can be orchestrated.

This way of orchestration allows the different instances of applications to share as much configuration files as possible, so that each process thinks that they bind to TCP ports (8030, 8040, 8050, etc), by taking advantage of Docker’s ability to map ports for publishing.

Below is an example of the docker compose file:

version: '3.6'
services:
  dapp1:
    image: docker.digihunch.com/dapp:${DAPP_VER}
    container_name: dapp1
    entrypoint: ["/opt/docker-entrypoint.sh","dapp"]
    ports:
      - 9301:8030          # BUSINESS SERVICE A
      - 9401:8040           # BUSINESS SERVICE B
      - 9501:8050           # BUSINESS SERVICE C
    mac_address: 2c:1f:4e:c5:9e:cf
    environment:
      - INSTANCE_TAG=dapp1 
      - MAX_JVM_HEAP=${DAPP_HEAP:-3892M}
    networks:
      - vcnet
    volumes:
      - /opt/dapp/etc:/opt/dapp/etc:ro
      - ./instances/dapp1/dapp.lic:/opt/dapp/etc/dapp.lic:ro
      - ./instances/dapp1/variables:/opt/dapp/etc/variables:ro
    deploy:
      resources:
        limits:
          cpus: '0.5'
          memory: ${DAPP_MEM:-4096M}
        reservations:
          memory: ${DAPP_MEM:-4096M}
    tty: true
  dapp2:
    image: docker.digihunch.com/dapp:${DAPP_VER}
    container_name: dapp2
    entrypoint: ["/opt/docker-entrypoint.sh","dapp"]
    ports:
      - 9302:8030          # BUSINESS SERVICE A
      - 9402:8040           # BUSINESS SERVICE B
      - 9502:8050           # BUSINESS SERVICE C
    mac_address: 2c:1f:4e:c5:9e:d0 
    environment:
      - INSTANCE_TAG=dapp2 
      - MAX_JVM_HEAP=${DAPP_HEAP:-3892M}
    networks:
      - vcnet
    volumes:
      - /opt/dapp/etc:/opt/dapp/etc:ro
      - ./instances/dapp2/dapp.lic:/opt/dapp/etc/dapp.lic:ro
      - ./instances/dapp2/variables:/opt/dapp/etc/variables:ro
    deploy:
      resources:
        limits:
          cpus: '0.5'
          memory: ${DAPP_MEM:-4096M}
        reservations:
          memory: ${DAPP_MEM:-4096M}
    tty: true
  dapp3:
    image: docker.digihunch.com/dapp:${DAPP_VER}
    container_name: dapp3
    entrypoint: ["/opt/docker-entrypoint.sh","dapp"]
    ports:
      - 9601:8030          # BUSINESS SERVICE A
      - 9602:8040           # BUSINESS SERVICE B
      - 9603:8050           # BUSINESS SERVICE C
    mac_address: 2c:1f:4e:c5:9e:d1 
    environment:
      - INSTANCE_TAG=dapp3 
      - MAX_JVM_HEAP=${DAPP_HEAP:-3892M}
    networks:
      - vcnet
    volumes:
      - /opt/dapp/etc:/opt/dapp/etc:ro
      - ./instances/dapp3/dapp.lic:/opt/dapp/etc/dapp.lic:ro
      - ./instances/dapp3/variables:/opt/dapp/etc/variables:ro
    deploy:
      resources:
        limits:
          cpus: '0.5'
          memory: ${DAPP_MEM:-4096M}
        reservations:
          memory: ${DAPP_MEM:-4096M}
    tty: true
networks:
  vcnet:
    driver: bridge
    driver_opts:
      com.docker.network.enable_ipv6: "false"

In this compose file, the environment variables are stored in .env file in the same directory and if they are not declared, the default is specified (syntax: ${VAR:-default}).

Helper scripts

The docker commands are fairly long so I had to organize them into several helper scripts. For example:

• docker-entrypoint.sh: this script is the ENTRYPOINT script for container. It is responsible for:
  • Initialization work that cannot be done in Dockerfile, such as setting environment variable
  • Launch the application, including pointing log file to stdout
  • Adding host entry for host.docker.internal to /etc/hosts, as a workaround to this issue with Docker on Linux
• build_image.sh: this script makes the image build process smoother
  • check if image to build already exist, and ask permission to delete the existing image if so;
  • build the image with Dockerfile, and create directory structure for Dockerfile to use during COPY instruction
• start_dapp_all.sh: this script starts all containers using docker-compose up and also add required iptables rules. We need to edit PREROUTING rules in IP tables to allow traffic between host NIC interface and the docker bridge interface, created each time service is up, as pointed out in previous post.
• stop_dapp_all.sh: this script removes the relevant iptables rules and stop all containers using docker-compose. Note that when deleting routing rules by number, start from the highest rule number and work your way down, since each deletion will cause the rules to be re-numbered.

Permission

The container uses a non-root user to run application (e.g. with su dhunch -c “command” from entry point script to run application as dhunch user), because the legacy application uses the same (non-root) user to perform its actions, and it is generally not advised to use root user. To ensure consistency, we need to create the dhunch user in container (in Dockerfile) so it’s uid and gid aligns with those of the host. The file and directory on the host to be access by the process in container also needs to allow dhunch user to read and write. Otherwise, entry point script will fail.

In the docker-compose file, we mount a file or a directory on the host to the container, and specify 😮 if it is read only mount, under volumes. We can alternatively use bind mount (check here for comparison). In either case, we need to keep in mind of the permission – owner alignment. For example, we have the following mount statement under volumes:

– /var/lib/dapp/dcontainer/archive:/var/lib/dapp/dhost/archive

We also need the entire directory hierarchy accessible to dhunch user. To configure this correctly, we need to create the entire directory hierarchy and set proper owner to it. Here is the comparison between the bad configuration and good configuration:

For application process running as dhunch, it also needs to write logs to stdout, so the result can be viewed from outside the container using docker logs command. The docker-entrypoint.sh script makes this happen by:

su dhunch -c "ln -sf /dev/stdout $DHUNCH_LOG_DIR/dhunch.log"

However, this command itself will run into permission issues. To fix, we need to add user dhunch to tty group (e.g. in Dockerfile as it’s needed on every container):

usermod -a -G tty dhunch

For application process to write to a shared volume on host (e.g. NFS), we can either allow access through volume mapping, or for performant access, mount the NFS share directly to container with proper driver.

Java application

For Java applications, only use the needed package (openjdk, openjdk-devel, openjdk-headless) as the Docker image size must be kept as small as possible. The headless package is for non-UI components, the devel package is for development stuff.

It is also worth-noting that the upper limit of heap size (Xmx) should be set based on the reserved memory of container (specified under docker-compose under resource limit and reservation). If heap is larger than container’s available memory, OOM will be triggered and the container will be killed. This article has some good explanation on this.