[BUG] It seems to run container with old image EVEN after rebuilding image with commands up --watch or watch with action 'rebuild'

[greyHairChooseLife]

Description

What I was doing

I was trying to M(ariadb)ERN stack and Continuous Deployment with github actions.

I'd like to run th is application with docker compose for both of dev and prod.

Before I found watch, I did like below:

• updating docker-compose.yml or env, installing new packages: docker compose down && docker compose up --build
• updating code lines that doesn't need compiling : set bind-mount

Now I'm trying watch or up --build --watch to avoid bind-mount. And as well I love the fact that everything is under develop key inside config file.

(+ Any difference between 'as-is' to 'to-be' on my purpose?)

So what is the problem?

The action: sync works great. Problem is action: rebuild. No matter which I command with, watch or up --build --watch.

It triggered well, rebuilt image well as I edit some file in path.(even without editing but only re-saving). But this recreated container with rebuilt image has the data from the old one. (It is true that both of container and img are brand new. I did watch -n 1 'docker images | grep')

For example, I added or removed query inside init.sql for dummy data but I can't find it, although it rebuilt img and recreated container. If I change env file that is listed for env_file: key inside config file, I can't find it as well inside newly created container.

Tree of project, commands, config file and other environments

In my project, there are now 2 services for DB, backend and I tried to rebuild both and they all same problem.

.
├── backend
│   ├── configs
│   ├── Dockerfile.dev
│   ├── index.js
│   └── ...
├── DB
│   ├── Dockerfile.dev
│   └── init.sql
└── dev-ops
    ├── docker-compose.dev.yml
    ├── env
    │   └── database.js
    └── commands.sh

# commands to run

docker compose --project-name $PROJECT_NAME --file $CONFIG_FILE watch
docker compose --project-name $PROJECT_NAME --file $CONFIG_FILE up --build --watch

# It works fine with old fashion one
docker compose --project-name $PROJECT_NAME down
docker compose --project-name $PROJECT_NAME --file $CONFIG_FILE up --build

# init.sql

USE my-devices;

CREATE TABLE devices (
  id INT PRIMARY KEY AUTO_INCREMENT,
  name VARCHAR(255) NOT NULL,
  price DECIMAL(10, 2) NOT NULL,
  created_at TIMESTAMP DEFAULT NOW()
);

-- dummy data
insert into devices (name, price) values ('iPhone SE', 250);
insert into devices (name, price) values ('iPhone SE2', 500);

# dockerfile for mariadb

# https://hub.docker.com/_/mariadb/tags
FROM mariadb:11.4.1-rc

COPY init.sql /docker-entrypoint-initdb.d/

docker-compose.dev.yml

services:
  dev-db:
    container_name: dev-db
    build:
      context: ../DB/
      dockerfile: Dockerfile.dev
      no_cache: true
    # command: 공식 이미지를 사용하는데, 이미지에 명세이 이미 있다.
    restart: unless-stopped
    env_file:
      - ./env/.env.DB.dev
    expose:
      - 3306
    develop:
      watch:
        - action: rebuild
          path: ../DB/
        - action: rebuild
          path: ./env/.env.DB.dev

  dev-backend:
    depends_on:
      - dev-db
    container_name: dev-backend
    build:
      context: ../backend/
      dockerfile: Dockerfile.dev
      no_cache: true
    command: npm run dev
    restart: unless-stopped
    environment:
      - NODE_ENV=development
    env_file:
      - ./env/.env.backend.dev
    ports:
      - 5001:5001
    # client server가 뚫리면 바꿔주자.
    # expose:
    #   - 5001
    # https://docs.docker.com/compose/file-watch/
    develop:
      watch:
        - action: sync
          path: ../backend/
          target: /app/
          ignore:
            - ../backend/node_modules
        - action: rebuild
          path: ./env/.env.backend.dev

Docker Environment

Client:
 Version:    26.1.1
 Context:    default
 Debug Mode: false
 Plugins:
  compose: Docker Compose (Docker Inc.)
    Version:  2.27.0
    Path:     /usr/lib/docker/cli-plugins/docker-compose

Server:
 Containers: 7
  Running: 6
  Paused: 0
  Stopped: 1
 Images: 193
 Server Version: 25.0.3
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: true
  Native Overlay Diff: false
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 7c3aca7a610df76212171d200ca3811ff6096eb8.m
 runc version: 
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.1.85-1-MANJARO
 Operating System: Manjaro Linux
 OSType: linux
 Architecture: x86_64
 CPUs: 24
 Total Memory: 31.12GiB
 Name: sy-230620
 ID: c1bb26b4-8c89-4a63-8c3c-70406282565c
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Username: [email protected]
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Just one more thing

There is one simple difference I found between two commands docker compose up --build --watch vs docker compose watch. It is "log".

# `watch` shows build logs as it rebuilding

Rebuilding service "dev-backend" after changes were detected...
[+] Building 4.1s (11/11) FINISHED  docker:default
 => [dev-backend internal] load build defini  0.0s
 => => transferring dockerfile: 196B          0.0s
 => [dev-backend internal] load metadata for  0.7s
 => [dev-backend internal] load .dockerignor  0.0s
 => => transferring context: 2B               0.0s
 => [dev-backend internal] load build contex  0.1s
 => => transferring context: 159.72kB         0.1s
 => [dev-backend 1/6] FROM docker.io/library  0.0s
 => CACHED [dev-backend 2/6] WORKDIR /app     0.0s
 => [dev-backend 3/6] RUN npm install -g pnp  1.2s
 => [dev-backend 4/6] COPY package.json pnpm  0.1s
 => [dev-backend 5/6] RUN pnpm install        1.6s 
 => [dev-backend 6/6] COPY . .                0.1s 
 => [dev-backend] exporting to image          0.3s 
 => => exporting layers                       0.3s 
 => => writing image sha256:6b6afb66561d8391  0.0s 
 => => naming to docker.io/library/dev-fbd-s  0.0s 
service "dev-backend" successfully built           

# `up --build --watch` shows just below

             ⦿ Rebuilding service "dev-backend" after changes were detected...
             ⦿ service "dev-backend" successfully built
dev-backend exited with code 143
dev-backend has been recreated

thanks for reading it and 우리 다 함께 행복해요~~~

[ndeloof]

If I change env file that is listed for env_file: key inside config file, I can't find it as well inside newly created container.

watch manages changes made in your code, but not changes in the compose file configuration (aka "compose model"). env_file content is parsed as you run the docker compose up.. command and won't be re-parsed later. This is a known limitation of compose watch feature.

I added or removed query inside init.sql for dummy data but I can't find it

Not sure I understand this as your example doesn't tell me how this init.sql is set and ran. Please note when container is re-created volumes from the previous one are kept (this is a general docker compose feature)

[greyHairChooseLife]

@ndeloof

Thank you for reply. I assumed that rebuilding means re-parsing whole part of 'watched' service in a comp
ose model'. And now I got it does not.

Not sure I understand this as your example doesn't tell me how this init.sql is set and ran. Please note when container is re-created volumes from the previous one are kept (this is a general docker compose feature)

I updated init.sql, Dockerfile file for the DB service. Would you please help me once again?

I didn't use volume, btw.

[ndeloof]

Your base image declares a volume: https://github.com/MariaDB/mariadb-docker/blob/master/11.4/Dockerfile#L137
so this one will "survive" a re-create. If you want to always start with fresh new data, you should set your init.sql file to drop tables before you inject initial data

There is one simple difference I found between two commands docker compose up --build --watch vs docker compose watch. It is "log".

docker compose watch should be reserved when you want to see sync/rebuild details in a separate console, while --watch is here to enable watch mode while getting application logs