Creating an Isolated Development Environment From Scratch Using Docker

Sometimes unit testing isn’t enough. Sometimes even running one of your applications locally isn’t enough. Occasionally we have to integration test our changes locally as developers.

A Simple Question

“Wouldn’t it be nice to execute one command and see our entire stack be deployed locally?” That’s the question that my boss recently asked me. And it planted a seed in my head that would bloom into something even greater than he’d initially intended.
I’m sure if you have been outside in the last couple years you’ve heard talk about Docker. “Docker is so great”, people rave, “have you tried compose? It fixed all our problems!”

At first I just wanted to ignore these people because their claims were unbelievable or if they were possible, they’d require a ton of time to realize. Then I got some hands on time, and I came to agree with them. It really is great if you put the time in.
Fast forward 18 months, we now have our entire stack running in containers with Swarm handling orchestration. And the experience of migrating our apps gave me the confidence to pull off what my boss asked.

The Inception

At a high level, our stack is essentially two PHP apps, one static HTML app, four Java apps, and a database. The PHP apps talk to one of the Java apps, and all of the apps aside from the HTML one interact with the database. So that’s where we’ll start.
I used the mysql docker image which can be downloaded from DockerHub. Then I added Flyway to prime our database on startup.

version: ‘3’
  services:
  mysql:
    image: mysql:5.7
    network_mode: “host”
    ports:
      – 3306:3306
  environment:
    MYSQL_ROOT_PASSWORD: password
    MYSQL_DATABASE: local_dev
  flyway:
    image: boxfuse/flyway
    hostname: flyway
    network_mode: “host”
    command: -url=jdbc:mysql://localhost/local_dev -user=root -password=password migrate
    volumes:
      – ./flyway/sql:/flyway/sql
    depends_on:
      – mysql

Bridge Networking

Next, I added the two PHP apps, and noticed that if the containers were using host networking they’d both try to bind port 80. So I switched to bridge networking, and this helped things.

version: ‘3’
services:
  mysql:
    image: mysql:5.7
    networks:
      – test
    ports: 
      – 3306:3306
    environment:
      MYSQL_ROOT_PASSWORD: password
      MYSQL_DATABASE: local_dev
  phpservice1:
    build: ./php-service-1/
    networks:
      – test
    ports: 
      – 8081:80
    volumes: 
      – ../php-service-1:/var/www/html/php-service-1
      – ./tmp/logs/php-service-1:/var/www/html/php-service-1/logs
    depends_on: – mysql
  phpservice2:
    build: ./php-service-2/
    networks:
      – test
    ports: 
      – 8082:80
    volumes: 
      – ../php-service-2:/var/www/html/php-service-2
      – ./tmp/logs/php-service-2:/var/www/html/php-service-2/logs
    depends_on: 
      – mysql
  flyway:
    image: boxfuse/flyway
    hostname: flyway
    networks: 
      – test
    command: 
       -url=jdbc:mysql://172.17.0.2/local_dev -user=root -password=password migrate
    volumes: 
       – ./flyway/sql:/flyway/sql
    depends_on: 
       – mysql
networks:
  test:
    driver: bridge

IP Addresses

However, the IP of the mysql instance wasn’t static, which made things unpredictable. I added the IPAM driver to my compose file, assigned static IPs to each service and reran without conflict. Both PHP apps were resolvable from the host at localhost:[port].

version: '3'
services:
  mysql:
    image: mysql:5.7
    networks:
      test:
        ipv4_address: 172.21.0.2
    ports:
      - 3306:3306
    environment:
      MYSQL_ROOT_PASSWORD: password
      MYSQL_DATABASE: local_dev
  phpservice1:
    build: ./php-service-1/
    networks:
      test:
        ipv4_address: 172.21.0.3
      ports:
        - 8082:80
    volumes:
      - ../php-service-1:/var/www/html/php-service-1
      - ./tmp/logs/php-service-1:/var/www/html/php-service-1/logs
    depends_on:
      - mysql
   #...
networks:
  test:
    driver: bridge
    ipam:
      config:
        - subnet: 172.21.0.0/20

Waiting For The Database

I next added the static HTML site which was straightforward. Then came the Java apps, which were a bit of a problem because they would launch before the database had data, and they did some schema validation on launch. I added the healthcheck command to deal with this, which actually forced me to downgrade to compose version 2.3.

version: ‘2.3’
services:
mysql:
image: mysql:5.7
networks:
test:
ipv4_address: 172.21.0.2
ports:
– 3306:3306
environment:
MYSQL_ROOT_PASSWORD: password
MYSQL_DATABASE: local_dev
healthcheck:
test: mysql –user=root –password=password -e “SHOW VARIABLES” >/dev/null 2>&1;
timeout: 20s
retries: 10
javaservice1:
build: ./java-service-1/
hostname: java-service-1
networks:
test:
ipv4_address: 172.21.0.7
ports:
– 8083:8080
volumes:
– ../tsoft_java-service-1/target/tsoft_java-service-1-0.1.0.jar:/opt/talksoft/java-service-1/jars/tsoft_java-service-1-0.1.0.jar
– ./tmp/logs/java-service-1:/opt/talksoft/java-service-1/logs
depends_on:
mysql:
condition: service_healthy
…

After this was done and we could deploy the whole stack with a single docker-compose up, I took things to the next step. I wanted a truly isolated environment but I also wanted to allow our applications to send SMTP messages but I wanted them captured by a fake server. Fakesmtp was the perfect tool for the job.

version: '2.3'
services:
...
fakesmtp:
image: digiplant/fake-smtp
hostname: fakesmtp
networks:
test:
ipv4_address: 172.21.0.11
ports:
- "25:25"
volumes:
- ./tmp/email:/var/mail

Innovating

Next, I wanted to be able to have our applications send text messages. For mocking a REST server I looked at two out of the box solutions: json-server and mock-server but neither provided the fine grained control I wanted.
We send text messages at scale and observe about a 40% response rate. So I wanted our mock RESTful server to be able to simulate this randomness as well as the randomness of the response content and the time to respond from when we sent the message.
This was just a proof of concept, but it actually went a lot easier than I thought.
First, I created an nginx reverse proxy server that would route traffic on 80 and 443 to our mock server. I then added a hosts file, with a few of the services we use aliased to the nginx server’s IP on our bridge network. Finally, I had the traffic that went to 80 and 443 on the nginx server forwarded to our mock server.

Nginx reverse proxy config

server {
listen 80;
listen 443 default ssl;
server_name localhost;
# ssl on;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_certificate cert.pem;
ssl_certificate_key cert.key;
location / {
proxy_pass http://172.21.0.12:8080;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-HTTPS 'True';
}
}

Hosts file

127.0.0.1 localhost
::1 localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
172.21.0.2 mysql
172.21.0.3 php-service-1
172.21.0.4 php-service-2
172.21.0.6 static-html
172.21.0.7 java-service-1
172.21.0.8 java-service-2
172.21.0.9 java-service-3
172.21.0.10 java-service-4
172.21.0.11 fakesmtp
172.21.0.12 mock-server
172.21.0.13 proxy-server api.twilio.com api.sms.voxox.com

Finally, adding the mock-server

version: '2.3'
services:
...
mockserver:
image: maven:3.5.3-jdk-8-alpine
hostname: mock-server
networks:
test:
ipv4_address: 172.21.0.12
ports:
- 8080:8080
working_dir: /usr/src/mymaven mvn clean install
entrypoint:
- java
- -jar
- /usr/src/mymaven/target/mock-server-0.1.0.jar
volumes:
- ./hosts:/etc/hosts
- ./mock-server/:/usr/src/mymaven

Conclusion and Source Code

This project was extremely satisfying, tying together Docker, proxies, restful APIs, devops, and yielding an isolated environment in which we can control expectations, simulate load, and develop flexibly while preserving or mocking the interdependencies of all our internal apps and even third-party ones.
You can see the final compose file below. For the full repo, please see here: https://github.com/JonasJSchreiber/local-development-scripts
(header image from fastcompany.com)

version: '2.3'
services:
mysql:
image: mysql:5.7
hostname: mysql
networks:
test:
ipv4_address: 172.21.0.2
ports:
- 3306:3306
environment:
MYSQL_ROOT_PASSWORD: password
MYSQL_DATABASE: local_dev
volumes:
- ./hosts:/etc/hosts
- ./mysql/mysql.cnf:/etc/mysql/conf.d/mysql.cnf
healthcheck:
test: mysql --user=root --password=password -e "SHOW VARIABLES" >/dev/null 2>&1;
timeout: 20s
retries: 10
phpservice1:
build: ./php-service-1/
hostname: php-service-1
networks:
test:
ipv4_address: 172.21.0.3
ports:
- 8082:80
volumes:
- ./hosts:/etc/hosts
- ../php-service-1:/var/www/html/php-service-1
- ./tmp/share:/opt/localuser/share
- ./tmp/logs/php-service-1:/var/www/html/php-service-1/logs
depends_on:
mysql:
condition: service_healthy
phpservice2:
build: ./php-service-2/
hostname: php-service-2
networks:
test:
ipv4_address: 172.21.0.4
ports:
- 8081:80
volumes:
- ./hosts:/etc/hosts
- ../php-service-1:/var/www/html/php-service-1
- ./tmp/share:/opt/localuser/share
- ./tmp/logs/php-service-1:/var/www/html/php-service-1/logs
depends_on:
mysql:
condition: service_healthy
flyway:
image: boxfuse/flyway
hostname: flyway
networks:
test:
ipv4_address: 172.21.0.5
command: -url=jdbc:mysql://mysql/local_dev -user=root -password=password migrate
volumes:
- ./hosts:/etc/hosts
- ./flyway/sql:/flyway/sql
depends_on:
mysql:
condition: service_healthy
statichtml:
build: ./static-html/
hostname: static-html
networks:
test:
ipv4_address: 172.21.0.6
ports:
- 8084:80
volumes:
- ./hosts:/etc/hosts
- ../web_mktg:/usr/local/apache2/htdocs/
javaservice1:
build: ./java-service-1/
hostname: java-service-1
networks:
test:
ipv4_address: 172.21.0.7
ports:
- 8083:8080
volumes:
- ./hosts:/etc/hosts
- ../java-service-1/target/java-service-1-0.1.0.jar:/opt/localuser/java-service-1/jars/java-service-1-0.1.0.jar
- ./tmp/share:/opt/localuser/share
- ./tmp/logs/java-service-1:/opt/localuser/java-service-1/logs
depends_on:
mysql:
condition: service_healthy
javaservice2:
build: ./java-service-2/
hostname: java-service-2
networks:
test:
ipv4_address: 172.21.0.8
volumes:
- ./hosts:/etc/hosts
- ../java-service-2//target/java-service-2-0.1.0.jar:opt/localuser/java-service-2/jars/java-service-2-0.1.0.jar
- ./tmp/share:/opt/localuser/share
- ./tmp/logs/java-service-2:/opt/localuser/java-service-2/logs
depends_on:
mysql:
condition: service_healthy
javaservice3:
build: ./java-service-3/
hostname: java-service-3
networks:
test:
ipv4_address: 172.21.0.9
volumes:
- ./hosts:/etc/hosts
- ../java-service-3/target/java-service-3-0.1.0.jar:/opt/localuser/java-service-3/jars/java-service-3-0.1.0.jar
- ./tmp/share:/opt/localuser/share
- ./tmp/logs/java-service-3:/opt/localuser/java-service-3/logs
depends_on:
mysql:
condition: service_healthy
javaservice4:
build: ./java-service-4/
hostname: java-service-4
networks:
test:
ipv4_address: 172.21.0.10
volumes:
- ./hosts:/etc/hosts
- ../java-service-4/target/java-service-4-0.1.0.jar:/opt/localuser/java-service-4/jars/java-service-4-0.1.0.jar
- ./tmp/logs/java-service-4:/opt/localuser/java-service-4/logs
depends_on:
mysql:
condition: service_healthy
fakesmtp:
image: digiplant/fake-smtp
hostname: fakesmtp
networks:
test:
ipv4_address: 172.21.0.11
ports:
- "25:25"
volumes:
- ./hosts:/etc/hosts
- ./tmp/email:/var/mail
mockserver:
image: maven:3.5.3-jdk-8-alpine
hostname: mock-server
networks:
test:
ipv4_address: 172.21.0.12
ports:
- 8080:8080
working_dir: /usr/src/mymaven mvn clean install
entrypoint:
- java
- -jar
- /usr/src/mymaven/target/mock-server-0.1.0.jar
volumes:
- ./hosts:/etc/hosts
- ./mock-server/:/usr/src/mymaven
proxyserver:
build: ./proxy-server/
hostname: proxy-server
networks:
test:
ipv4_address: 172.21.0.13
volumes:
- ./hosts:/etc/hosts
- ./proxy-server/proxy.conf:/etc/nginx/conf.d/default.conf
- ./proxy-server/ssl/cert.key:/etc/nginx/cert.key
- ./proxy-server/ssl/cert.pem:/etc/nginx/cert.pem
ports:
- 443:443
- 80:80
networks:
test:
driver: bridge
ipam:
config:
- subnet: 172.21.0.0/20