Plausible is fetching MaxMind GeoIP databases too often

[perfectly-preserved-pie]

I just got this email from Maxmind:

Dear MaxMind Customer,
Your account has reached the daily limit for database downloads. Any additional download attempts today from your account will fail. Learn more about database download limits.
To avoid download errors we recommend that you limit your downloads of each database to no more than once per day per server. You can sign in to your account and check your GeoIP download history for information on IP addresses you are downloading databases from.
To ensure that you are downloading your databases efficiently, you may consult the update schedule for GeoIP databases. If you have any questions, please contact us at [email protected].
Sincerely,
The Team at MaxMind

And looking at my download history in my MaxMind account, it seems Plausible is fetching the same exact database quite frequently:

This frequency is unnecessary because MaxMind updates their databases on a regular schedule, every Tuesday and Friday.

Could there be a new environment variable added, perhaps MAXMIND_UPDATE_FREQUENCY, that we could use to dictate when Plausible should fetch a new database from MaxMind to avoid hitting the download limits of a free account?

Or, does anyone have other suggestions to work around this? Could this be a problem with my installation?

[ruslandoga]

👋 @perfectly-preserved-pie

Could you please check your docker compose logs to see if Plausible is restarting?

[perfectly-preserved-pie]

Unfortunately it has started to crash and restart over and over again :(

The clickhouse and postgres containers are fine; no errors in there. But Plausible logs the messages above and keeps quitting and restarting.

[ruslandoga]

@perfectly-preserved-pie please post full logs for the last session (when it was going smoothly at first and then started to have problems) for all services (including PostgreSQL and ClickHouse). And also please post your docker-compose.yml and describe your host (OS, Docker version, anything else relevant).

[ruslandoga]

@madanyang we use locus for geo lookups. Here's the description of the DB update flow https://github.com/g-andrade/locus#remote-sources-downloading-and-updating If you are not experiencing 429 errors, I don't think changing update_period makes sense yet.

[madanyang]

@ruslandoga now at least I know the where this 6 hr interval is coming. While I agree if aint' broke don't fix it approach, the same approach also prevents user controllable variables like update frequency which was available when using maxmind docker. So for me this is a feature request.

[perfectly-preserved-pie]

@ruslandoga

Plausible:

Starting dependencies..
Starting repos..
create Plausible.Repo database if it doesn't exist
create Plausible.IngestRepo database if it doesn't exist
Creation of Db successful!
Loading plausible..
Starting dependencies..
Starting repos..
Running migrations for Elixir.Plausible.Repo
Running migrations for Elixir.Plausible.IngestRepo
Migrations successful!
05:00:08.985 [error] GenServer Plausible.Site.Cache.Warmer.RecentlyUpdated terminating
** (DBConnection.ConnectionError) connection not available and request was dropped from queue after 2701ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:

  1. Ensuring your database is available and that you can connect to it
  2. Tracking down slow queries and making sure they are running fast enough
  3. Increasing the pool_size (although this increases resource consumption)
  4. Allowing requests to wait longer by increasing :queue_target and :queue_interval

See DBConnection.start_link/2 for more information

    (ecto_sql 3.9.2) lib/ecto/adapters/sql.ex:913: Ecto.Adapters.SQL.raise_sql_call_error/1
    (ecto_sql 3.9.2) lib/ecto/adapters/sql.ex:828: Ecto.Adapters.SQL.execute/6
    (ecto 3.9.5) lib/ecto/repo/queryable.ex:229: Ecto.Repo.Queryable.execute/4
    (ecto 3.9.5) lib/ecto/repo/queryable.ex:19: Ecto.Repo.Queryable.all/3
    (plausible 0.0.1) lib/plausible/site/cache.ex:220: anonymous fn/2 in Plausible.Site.Cache.refresh/3
    (plausible 0.0.1) lib/plausible/site/cache.ex:235: Plausible.Site.Cache.time_it/1
    (plausible 0.0.1) lib/plausible/site/cache.ex:228: Plausible.Site.Cache.measure_duration/2
    (plausible 0.0.1) lib/plausible/site/cache/warmer.ex:62: Plausible.Site.Cache.Warmer.handle_cycle/1
Last message: :"$cycle"
05:00:08.987 [error] Handler "Elixir.OpentelemetryOban.PluginHandler.plugin_exception" has failed and has been detached. Class=:error
Reason=:function_clause
Stacktrace=[
  {OpentelemetryOban.PluginHandler, :handle_plugin_exception,
   [
     [:oban, :plugin, :exception],
     %{duration: 2460541415, monotonic_time: -576458747475359555},
     %{
       conf: %Oban.Config{
         dispatch_cooldown: 5,
         engine: Oban.Queue.BasicEngine,
         get_dynamic_repo: nil,
         log: false,
         name: Oban,
         node: "plausible@plausible",
         notifier: Oban.Notifiers.Postgres,
         peer: Oban.Peers.Postgres,
         plugins: [
           {Oban.Plugins.Stager, [interval: 5000]},
           {Oban.Plugins.Lifeline, [rescue_after: 7200000]},
           {Oban.Plugins.Cron,
            [
              crontab: [
                {"0 0 * * *", Plausible.Workers.RotateSalts},
                {"0 * * * *", Plausible.Workers.ScheduleEmailReports},
                {"0 * * * *", Plausible.Workers.SendSiteSetupEmails},
                {"0 12 * * *", Plausible.Workers.SendCheckStatsEmails},
                {"*/15 * * * *", Plausible.Workers.SpikeNotifier},
                {"0 0 * * *", Plausible.Workers.CleanEmailVerificationCodes},
                {"0 1 * * *", Plausible.Workers.CleanInvitations},
                {"0 */2 * * *", Plausible.Workers.ExpireDomainChangeTransitions}
              ]
            ]},
           {Oban.Plugins.Pruner, [max_age: 2592000]}
         ],
         prefix: "public",
         queues: [
           rotate_salts: [limit: 1],
           schedule_email_reports: [limit: 1],
           send_email_reports: [limit: 1],
           spike_notifications: [limit: 1],
           check_stats_emails: [limit: 1],
           site_setup_emails: [limit: 1],
           clean_email_verification_codes: [limit: 1],
           clean_invitations: [limit: 1],
           google_analytics_imports: [limit: 1],
           domain_change_transition: [limit: 1]
         ],
         repo: Plausible.Repo,
         shutdown_grace_period: 15000,
         testing: :disabled
       },
       kind: :error,
       plugin: Oban.Plugins.Pruner,
       reason: %DBConnection.ConnectionError{
         message: "connection not available and request was dropped from queue after 2460ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:\n\n  1. Ensuring your database is available and that you can connect to it\n  2. Tracking down slow queries and making sure they are running fast enough\n  3. Increasing the pool_size (although this increases resource consumption)\n  4. Allowing requests to wait longer by increasing :queue_target and :queue_interval\n\nSee DBConnection.start_link/2 for more information\n",
         severity: :error,
         reason: :queue_timeout
       },
       stacktrace: [
         {DBConnection, :transaction, 3,
          [
            file: 'lib/db_connection.ex',
            line: 972,
            error_info: %{module: Exception}
          ]},
         {Oban.Plugins.Pruner, :"-handle_info/2-fun-0-", 2,
          [file: 'lib/oban/plugins/pruner.ex', line: 113]},
         {:telemetry, :span, 3,
          [file: '/app/deps/telemetry/src/telemetry.erl', line: 321]},
         {Oban.Plugins.Pruner, :handle_info, 2,
          [file: 'lib/oban/plugins/pruner.ex', line: 112]},
         {:gen_server, :try_dispatch, 4, [file: 'gen_server.erl', line: 1123]},
         {:gen_server, :handle_msg, 6, [file: 'gen_server.erl', line: 1200]},
         {:proc_lib, :init_p_do_apply, 3, [file: 'proc_lib.erl', line: 240]}
       ],
       telemetry_span_context: #Reference<0.1026890792.1167065116.158354>
     },
     []
   ], [file: 'lib/opentelemetry_oban/plugin_handler.ex', line: 52]},
  {:telemetry, :"-execute/3-fun-0-", 4,
   [file: '/app/deps/telemetry/src/telemetry.erl', line: 160]},
  {:lists, :foreach_1, 2, [file: 'lists.erl', line: 1442]},
  {:telemetry, :span, 3,
   [file: '/app/deps/telemetry/src/telemetry.erl', line: 333]},
  {Oban.Plugins.Pruner, :handle_info, 2,
   [file: 'lib/oban/plugins/pruner.ex', line: 112]},
  {:gen_server, :try_dispatch, 4, [file: 'gen_server.erl', line: 1123]},
  {:gen_server, :handle_msg, 6, [file: 'gen_server.erl', line: 1200]},
  {:proc_lib, :init_p_do_apply, 3, [file: 'proc_lib.erl', line: 240]}
]

05:00:08.989 [error] Handler "Elixir.OpentelemetryOban.PluginHandler.plugin_exception" has failed and has been detached. Class=:error
Reason=:function_clause
Stacktrace=[
  {OpentelemetryOban.PluginHandler, :handle_plugin_exception,
   [
     [:oban, :plugin, :exception],
     %{duration: 1174858745, monotonic_time: -576458747475061339},
     %{
       conf: %Oban.Config{
         dispatch_cooldown: 5,
         engine: Oban.Queue.BasicEngine,
         get_dynamic_repo: nil,
         log: false,
         name: Oban,
         node: "plausible@plausible",
         notifier: Oban.Notifiers.Postgres,
         peer: Oban.Peers.Postgres,
         plugins: [
           {Oban.Plugins.Stager, [interval: 5000]},
           {Oban.Plugins.Lifeline, [rescue_after: 7200000]},
           {Oban.Plugins.Cron,
            [
              crontab: [
                {"0 0 * * *", Plausible.Workers.RotateSalts},
                {"0 * * * *", Plausible.Workers.ScheduleEmailReports},
                {"0 * * * *", Plausible.Workers.SendSiteSetupEmails},
                {"0 12 * * *", Plausible.Workers.SendCheckStatsEmails},
                {"*/15 * * * *", Plausible.Workers.SpikeNotifier},
                {"0 0 * * *", Plausible.Workers.CleanEmailVerificationCodes},
                {"0 1 * * *", Plausible.Workers.CleanInvitations},
                {"0 */2 * * *", Plausible.Workers.ExpireDomainChangeTransitions}
              ]
            ]},
           {Oban.Plugins.Pruner, [max_age: 2592000]}
         ],
         prefix: "public",
         queues: [
           rotate_salts: [limit: 1],
           schedule_email_reports: [limit: 1],
           send_email_reports: [limit: 1],
           spike_notifications: [limit: 1],
           check_stats_emails: [limit: 1],
           site_setup_emails: [limit: 1],
           clean_email_verification_codes: [limit: 1],
           clean_invitations: [limit: 1],
           google_analytics_imports: [limit: 1],
           domain_change_transition: [limit: 1]
         ],
         repo: Plausible.Repo,
         shutdown_grace_period: 15000,
         testing: :disabled
       },
       kind: :error,
       plugin: Oban.Plugins.Stager,
       reason: %DBConnection.ConnectionError{
         message: "connection not available and request was dropped from queue after 1174ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:\n\n  1. Ensuring your database is available and that you can connect to it\n  2. Tracking down slow queries and making sure they are running fast enough\n  3. Increasing the pool_size (although this increases resource consumption)\n  4. Allowing requests to wait longer by increasing :queue_target and :queue_interval\n\nSee DBConnection.start_link/2 for more information\n",
         severity: :error,
         reason: :queue_timeout
       },
       stacktrace: [
         {DBConnection, :transaction, 3,
          [
            file: 'lib/db_connection.ex',
            line: 972,
            error_info: %{module: Exception}
          ]},
         {Oban.Plugins.Stager, :"-handle_info/2-fun-0-", 2,
          [file: 'lib/oban/plugins/stager.ex', line: 99]},
         {:telemetry, :span, 3,
          [file: '/app/deps/telemetry/src/telemetry.erl', line: 321]},
         {Oban.Plugins.Stager, :handle_info, 2,
          [file: 'lib/oban/plugins/stager.ex', line: 98]},
         {:gen_server, :try_dispatch, 4, [file: 'gen_server.erl', line: 1123]},
         {:gen_server, :handle_msg, 6, [file: 'gen_server.erl', line: 1200]},
         {:proc_lib, :init_p_do_apply, 3, [file: 'proc_lib.erl', line: 240]}
       ],
       telemetry_span_context: #Reference<0.1026890792.1167065116.158358>
     },
     []
   ], [file: 'lib/opentelemetry_oban/plugin_handler.ex', line: 52]},
  {:telemetry, :"-execute/3-fun-0-", 4,
   [file: '/app/deps/telemetry/src/telemetry.erl', line: 160]},
  {:lists, :foreach_1, 2, [file: 'lists.erl', line: 1442]},
  {:telemetry, :span, 3,
   [file: '/app/deps/telemetry/src/telemetry.erl', line: 333]},
  {Oban.Plugins.Stager, :handle_info, 2,
   [file: 'lib/oban/plugins/stager.ex', line: 98]},
  {:gen_server, :try_dispatch, 4, [file: 'gen_server.erl', line: 1123]},
  {:gen_server, :handle_msg, 6, [file: 'gen_server.erl', line: 1200]},
  {:proc_lib, :init_p_do_apply, 3, [file: 'proc_lib.erl', line: 240]}
]

Clickhouse:

Merging configuration file '/etc/clickhouse-server/config.d/docker_related_config.xml'.
Logging trace to /var/log/clickhouse-server/clickhouse-server.log
Logging errors to /var/log/clickhouse-server/clickhouse-server.err.log
Processing configuration file '/etc/clickhouse-server/config.xml'.
Merging configuration file '/etc/clickhouse-server/config.d/docker_related_config.xml'.
Saved preprocessed configuration to '/var/lib/clickhouse/preprocessed_configs/config.xml'.
Processing configuration file '/etc/clickhouse-server/users.xml'.
Saved preprocessed configuration to '/var/lib/clickhouse/preprocessed_configs/users.xml'.

Postgres:

PostgreSQL Database directory appears to contain a database; Skipping initialization

2024-04-29 04:26:28.748 UTC [1] LOG:  starting PostgreSQL 12.18 on x86_64-pc-linux-musl, compiled by gcc (Alpine 13.2.1_git20231014) 13.2.1 20231014, 64-bit
2024-04-29 04:26:28.748 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
2024-04-29 04:26:28.748 UTC [1] LOG:  listening on IPv6 address "::", port 5432
2024-04-29 04:26:28.752 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2024-04-29 04:26:28.772 UTC [22] LOG:  database system was shut down at 2024-04-29 04:26:16 UTC
2024-04-29 04:26:28.780 UTC [1] LOG:  database system is ready to accept connections
2024-04-29 08:01:03.435 UTC [38] LOG:  could not receive data from client: Connection reset by peer
2024-04-29 08:01:03.435 UTC [39] LOG:  could not receive data from client: Connection reset by peer
2024-04-29 08:01:03.435 UTC [40] LOG:  could not receive data from client: Connection reset by peer
2024-04-29 08:01:03.435 UTC [41] LOG:  could not receive data from client: Connection reset by peer
2024-04-29 08:01:03.435 UTC [42] LOG:  could not receive data from client: Connection reset by peer
2024-04-29 08:01:03.435 UTC [43] LOG:  could not receive data from client: Connection reset by peer
2024-04-29 08:01:03.435 UTC [44] LOG:  could not receive data from client: Connection reset by peer
2024-04-29 08:01:03.435 UTC [45] LOG:  could not receive data from client: Connection reset by peer
2024-04-29 08:01:03.435 UTC [46] LOG:  could not receive data from client: Connection reset by peer
2024-04-29 08:01:03.439 UTC [47] LOG:  could not receive data from client: Connection reset by peer

docker-compose.yml:

version: "3.3"
services:
  mail:
    hostname: plausible_smtp
    image: bytemark/smtp
    restart: always

  plausible_db:
    hostname: plausible_postgres
    # supported versions are 12, 13, and 14
    image: postgres:12-alpine
    restart: always
    volumes:
       - /appdata/plausible/postgres/db-data:/var/lib/postgresql/data
    environment:
      - POSTGRES_PASSWORD=redacted

  plausible_events_db:
    hostname: plausible_clickhouse
    image: clickhouse/clickhouse-server:23.3.7.5-alpine
    restart: always
    volumes:
      - /appdata/plausible/clickhouse/event-data:/var/lib/clickhouse
      - /appdata/plausible/clickhouse/logs:/var/log/clickhouse-server
      - ./clickhouse/clickhouse-config.xml:/etc/clickhouse-server/config.d/logging.xml:ro
      - ./clickhouse/clickhouse-user-config.xml:/etc/clickhouse-server/users.d/logging.xml:ro
    ulimits:
      nofile:
        soft: 262144
        hard: 262144

  plausible:
    hostname: plausible
    image: plausible/analytics:v2.0
    restart: always
    command: sh -c "sleep 10 && /entrypoint.sh db createdb && /entrypoint.sh db migrate && /entrypoint.sh run"
    depends_on:
      - plausible_db
      - plausible_events_db
      - mail
    #ports:
     # - 8000:8000
    volumes:
      - /appdata/plausible/cities/geonames.csv:/etc/app/cities/geonames.csv:ro
    env_file:
      - plausible-conf.env

volumes:
  db-data:
    driver: local
  event-data:
    driver: local

networks:
  default:
    external:
      name: theusual

I'm running Ubuntu 22.04 LTS and Docker version 26.1.0, build 9714adc.

[rafammpp]

I got the same email yesterday. It's because I have a second server for making queries, but that server is only active on demand. So, every time it wakes up, it downloads another db from maxmind. At least, it should be limited to only download a db once per day, not every time plausible starts.

[perfectly-preserved-pie]

Just following up here... the problem went away after a few days with no changes to the config. Only other thing I can think of is that I restarted the host.

Gonna blame gremlins for this one 🤷‍♂️