Go to file
Adam Veldhousen 70d300df5e
ci.vdhsn.com/push Build was killed Details
add catawiki catalog sync, DX improvements, ci
- drone ci build using kaniko

This builds images extremely slowly, I've implemented some custom fixes to the
plugin to improve performance but it's still slow to build images. I will
try to implement a DinD solution.
There are issues in kaniko about this: https://github.com/GoogleContainerTools/kaniko/issues/875

Squashed commit of the following:

commit 1fd65ad139
Author: Adam Veldhousen <adamveld12@gmail.com>
Date:   Sat Jul 15 18:17:55 2023 -0500

    add deploy script

commit 075e15f218
Author: Adam Veldhousen <adamveld12@gmail.com>
Date:   Sat Jul 15 17:53:24 2023 -0500

    bump limits

commit 2d45234e7b
Author: Adam Veldhousen <adamveld12@gmail.com>
Date:   Sat Jul 15 17:51:19 2023 -0500

    sync button in tilt, support syncing all targets simultaneously

commit 75e73a1171
Author: Adam Veldhousen <adamveld12@gmail.com>
Date:   Sat Jul 15 17:03:50 2023 -0500

    create admin user on startup

commit 40cf74560d
Author: Adam Veldhousen <adamveld12@gmail.com>
Date:   Fri Jul 14 11:14:00 2023 -0500

    test more cache

commit 1fbbfe7548
Author: Adam Veldhousen <adamveld12@gmail.com>
Date:   Fri Jul 14 11:11:34 2023 -0500

    test tmpfs

commit 0f48c098d9
Author: Adam Veldhousen <adamveld12@gmail.com>
Date:   Fri Jul 14 11:08:32 2023 -0500

    add tmpfs

commit d37142cfef
Author: Adam Veldhousen <adamveld12@gmail.com>
Date:   Fri Jul 14 11:02:39 2023 -0500

    crank verbosity

commit 178eb5bc38
Author: Adam Veldhousen <adamveld12@gmail.com>
Date:   Fri Jul 14 10:44:08 2023 -0500

    bump requests

commit b0aa0b39f0
Author: Adam Veldhousen <adamveld12@gmail.com>
Date:   Fri Jul 14 10:40:04 2023 -0500

    test fewer concurrent builds

commit 221e378a53
Author: Adam Veldhousen <adamveld12@gmail.com>
Date:   Thu Jul 13 20:40:18 2023 -0500

    test

commit 6c03914341
Author: Adam Veldhousen <adamveld12@gmail.com>
Date:   Thu Jul 13 19:01:22 2023 -0500

    snapshot mode for faster build

commit 91c56d430e
Author: Adam Veldhousen <adamveld12@gmail.com>
Date:   Wed Jul 12 22:40:12 2023 -0500

    try script

commit 4a119c2523
Author: Adam Veldhousen <adamveld12@gmail.com>
Date:   Wed Jul 12 22:28:41 2023 -0500

    test promote

commit b1062a4001
Author: Adam Veldhousen <adamveld12@gmail.com>
Date:   Wed Jul 12 22:26:58 2023 -0500

    test

commit c06804dfe6
Author: Adam Veldhousen <adamveld12@gmail.com>
Date:   Wed Jul 12 17:47:48 2023 -0500

    beta promote and ignore

commit 643e104c37
Author: Adam Veldhousen <adamveld12@gmail.com>
Date:   Wed Jul 12 16:21:39 2023 -0500

    tweakies

commit 478e65ac70
Author: Adam Veldhousen <adamveld12@gmail.com>
Date:   Wed Jul 12 16:09:54 2023 -0500

    un woops

commit 966d48c193
Author: Adam Veldhousen <adamveld12@gmail.com>
Date:   Wed Jul 12 15:23:08 2023 -0500

    woops

commit 2829743d17
Author: Adam Veldhousen <adamveld12@gmail.com>
Date:   Wed Jul 12 15:22:03 2023 -0500

    test internal routing

commit e08e68b8ad
Author: Adam Veldhousen <adamveld12@gmail.com>
Date:   Wed Jul 12 14:07:31 2023 -0500

    push images

commit 5705bca068
Author: Adam Veldhousen <adamveld12@gmail.com>
Date:   Wed Jul 12 13:40:48 2023 -0500

    ugh

commit 1699b08d5f
Author: Adam Veldhousen <adamveld12@gmail.com>
Date:   Wed Jul 12 13:39:49 2023 -0500

    fixup client builds

commit 9ec8616ae1
Author: Adam Veldhousen <adamveld12@gmail.com>
Date:   Wed Jul 12 13:36:36 2023 -0500

    update go mods

commit 163b25fd67
Author: Adam Veldhousen <adamveld12@gmail.com>
Date:   Wed Jul 12 02:11:02 2023 -0500

    fix fronend builds

commit e5cc7d5c79
Author: Adam Veldhousen <adamveld12@gmail.com>
Date:   Wed Jul 12 01:56:08 2023 -0500

    resources

commit 8088b95c1e
Author: Adam Veldhousen <adamveld12@gmail.com>
Date:   Wed Jul 12 01:38:21 2023 -0500

    big test

commit 38fef91c49
Author: Adam Veldhousen <adamveld12@gmail.com>
Date:   Wed Jul 12 01:26:09 2023 -0500

    testicles

commit 53baa0a647
Author: Adam Veldhousen <adamveld12@gmail.com>
Date:   Wed Jul 12 01:24:50 2023 -0500

    testicles

commit 16901ec5bb
Author: Adam Veldhousen <adamveld12@gmail.com>
Date:   Wed Jul 12 01:23:38 2023 -0500

    testicles

commit 3478d4f512
Author: Adam Veldhousen <adamveld12@gmail.com>
Date:   Wed Jul 12 01:19:58 2023 -0500

    test 2

commit 16298ada78
Author: Adam Veldhousen <adamveld12@gmail.com>
Date:   Wed Jul 12 01:13:53 2023 -0500

    more test

commit 80524a1d06
Author: Adam Veldhousen <adamveld12@gmail.com>
Date:   Wed Jul 12 01:09:03 2023 -0500

    test

commit b9916204de
Author: Adam Veldhousen <adamveld12@gmail.com>
Date:   Wed Jul 12 00:19:41 2023 -0500

    tools image

commit 6e414b3a1b
Author: Adam Veldhousen <adamveld12@gmail.com>
Date:   Wed Jul 12 00:00:23 2023 -0500

    test

commit abb6183e88
Author: Adam Veldhousen <adamveld12@gmail.com>
Date:   Tue Jul 11 22:55:58 2023 -0500

    added catawiki scrape

commit 2643817e93
Author: Adam Veldhousen <adamveld12@gmail.com>
Date:   Tue Jul 11 14:40:05 2023 -0500

    add catawiki syncer
2023-07-15 18:18:56 -05:00
.vscode MVP build (#1) 2023-06-12 20:25:55 -05:00
doc auth deployment 2023-07-10 01:52:02 -05:00
env auth kustomize 2023-07-10 03:46:29 -05:00
hack add catawiki catalog sync, DX improvements, ci 2023-07-15 18:18:56 -05:00
src add catawiki catalog sync, DX improvements, ci 2023-07-15 18:18:56 -05:00
.drone.yml add catawiki catalog sync, DX improvements, ci 2023-07-15 18:18:56 -05:00
.gitignore MVP build (#1) 2023-06-12 20:25:55 -05:00
.sops.yaml MVP build (#1) 2023-06-12 20:25:55 -05:00
.tool-versions MVP build (#1) 2023-06-12 20:25:55 -05:00
LICENSE docs and architecture planning 2023-04-16 13:06:10 -05:00
Makefile add catawiki catalog sync, DX improvements, ci 2023-07-15 18:18:56 -05:00
README.md add catawiki catalog sync, DX improvements, ci 2023-07-15 18:18:56 -05:00
Tiltfile add catawiki catalog sync, DX improvements, ci 2023-07-15 18:18:56 -05:00
go.work add catawiki catalog sync, DX improvements, ci 2023-07-15 18:18:56 -05:00
go.work.sum MVP build (#1) 2023-06-12 20:25:55 -05:00

README.md

Barretthousen

Build Status Search and get alerts for items across the most popular auction sites.

Built with microservice architecture and various fancy tools for learning purposes

Technology

This project is for fun and learning so there are a lot of tools at play:

Infra

  • Tilt: local development env orchestration
  • Ctlptl: Automate local kubernetes clusters for local dev
  • Docker
  • Postgres:
  • Traefik:
  • Kustomize:
  • Helm:

Backend

  • Go
  • goose: Go SQL db migration tool
  • Sqlc: Generate Go code to query SQL databases from migrations in a type safe way
  • Buf: Nice frontend for protoc and .proto libraries
  • Protobufs:

Frontend

Diagram

Services

Auction terms glossary

  • Web-client Frontend site that users access to browse upcoming auctions and subscribe for updates to their searches

  • Admin-client Enables site administrators to kicks off scrape and email jobs in Runner and Ringman, and view status on current/past jobs.

  • Catalog API for searching upcoming auctions.

  • Runner Scrapes sites for upcoming auctions. Transforms the data into a suitable shape and stores it for use by the frontend.

  • Ringman Manages email subscriptions and sending emails to users about upcoming auctions.

  • Proxy-web Proxies JSON HTTP API requests for the web client to GRPC based services downstream

  • Proxy-admin Proxies JSON HTTP API requests for the admin client to GRPC based services downstream

Contributing

You will need the following prereqs installed:

Everything you need is in the makefile:

# install asdf tools, sqlc, buf
make setup

# generate protobufs and sql boilerplate
make gen

# spin up a k8s cluster, build and deploy services locally w/ hot reloading - be patient this takes a few minutes first run
make dev

# build production docker images for the backend microservices, optionally push to the respective env
make build-backend-image SERVICE=[catalog, runner, proxy-web, proxy-admin] [ENV=[beta, prod]]

# build client docker image for web frontends, optionally push to the respective env
make build-client-image SERVICE=[web-client, admin-client] [ENV=[beta, prod]]

# acceptance tests
make acceptance-test SERVICE=[runner,catalog] ORIGIN=[beta.barretthousen.com,beta.admin.barretthousen.com]

For how to deploy read me.