It's Not 'Click and Done' from the Start: Building a Data Environment with AI

Can't Leave AI Out Even If I Wanted To

These days, no matter where you look—LinkedIn, news, social media—you can't escape AI. It's everyone's talking point, and that's just reality. To be honest, there's so much chatter and things change overnight that it does make my head spin a little.

On a slight tangent... I personally have a bit of an indie sensibility, so I haven't even watched Avatar or Squid Game, and I work at a startup (?). So seeing Attention, GPT, and RL—topics I used to secretly study in the shadows with folks a few years ago—suddenly become incredibly mainstream makes me feel like I'm watching my favorite underground indie singer suddenly hit the big time...

Getting back to the point... Recently, while building a new analytics system for DelightRoom's DARO service and operating the data environment, I was able to boost productivity significantly by brainstorming and collaborating with Claude Code, and I'd like to share what I gained from that process. If you know AI is good but are struggling to find practical use cases, I hope this helps.

Everyone's Busy, and I'm Still Poking at Prod Data...

DARO, DelightRoom's ad monetization solution business model, has shown steep revenue growth since its launch in '25. Because it's such a fast-growing product, prioritizing the setup of a stable data environment naturally got pushed back. As the person in charge of data, I also focused heavily on metric growth, thinking, "Creating value in the market comes first."

As a result, like many analysts in the early stages of a product, I naturally ended up running queries directly against the production DB for data analysis, which eventually reached its limits.

• I always carried the risk of causing an issue by firing a bad query at the prod DB, and

• With flexible data pipelining and orchestration being impossible, setting up data marts was difficult, and getting fast query results became even harder as the data grew larger.

Ultimately, I was growing more distant from the data, making it hard to poke around and explore different areas.

As data sources and clients multiplied, we found ourselves handling TB-scale data daily. The DARO domain couldn't ignore these issues anymore, so we set out to build a proper analytics environment.

Basic MDW Architecture

Since DARO was collecting massive amounts of diverse data (SDK, SSP, CPS, etc.) via our own Kafka and 3rd party Report APIs, we basically built an S3-based Data Lake and layered Athena on top of it.

On top of that, an engineer laid the initial groundwork to handle orchestration using our existing Airflow and DBT setup.

Since resources are always tight, and the thirsty dig the well, I ended up handling the actual Data Pipelining & Modeling and operational tasks myself. I was able to streamline a lot of it with Claude Code.

Data Pipelining & Modeling

At its core, I used Claude Code's skill feature to automate the repetitive parts of pipeline creation and modeling. I created and used two main skills: one for connecting sources, and another for generating data tables afterward, keeping them separated.

/add-dbt-daro-source

  - S3 데이터를 Athena 외부 테이블로 등록하는 커맨드
  - S3 경로 탐색 → 샘플 파일 다운로드 → 스키마(컬럼/타입) 자동 추출
  - dbt_project.yml에 partition projection 설정 추가 (dt 파티션 기반)
  - AWS Glue에 외부 테이블 생성 (create_external_table 매크로 실행)
  - daro_s3.yml에 source 정의 등록 (컬럼 설명 포함)
  - Athena 쿼리 + dbt show로 데이터 접근 검증

The first step in building the analytics environment was connecting the source files in S3 to Athena. Creating YAML files individually to connect over 50 sources was just too much manual work.

I started the automation by simply asking Claude to turn the initial connection process into a skill, and I was able to improve its stability and usability through the following tweaks.

• Since data samples fetched for schema exploration are in JSON format, columns could be missing, so I set it up to explore multiple data samples.

• Instead of auto-generating table names, it uses the table names specified by the user.

• Configured it so source tables are connected to both dev and prod environments simultaneously.

/add-dbt-daro-model                                                 
                                                                                                                                                                                                                    
  - dbt 모델 생성 커맨드 (staging / mart / dimension)                 
  - 인터랙티브하게 모델 타입, materialization 전략, 소스 테이블, grain, unique key 등을 질문하며 진행
  - 답변 기반으로 SQL 파일 + YAML 문서(컬럼 설명, 테스트)를 자동 생성
  - 생성 후 dbt compile → dbt run → dbt test까지 검증
  - DARO OLAP 원칙 적용: single-partition, partition pruning, on_schema_change='fail'

For the second stage, table design, I had to account for the reality that actual business logic involves ambiguity. So, rather than letting Claude make unilateral decisions, I designed it to finalize the product specs through as much user interaction as possible.

First, you choose whether you're creating a staging, mart, or dimension data model, and define the necessary columns together. Then, it performs base modeling on top of the existing architecture, followed by compiling and testing.

• In this stage, it was necessary to separately inject the specific quirks of staging tables (deduplication, null column handling, etc.), and

• referencing existing queries to maintain a consistent style was key.

Data Governance / Cataloging

Additionally, while working on this, another area where I truly felt the value of AI was documentation.

Within DelightRoom's data group, we have a predefined policy to ensure governance when building data marts. We also have a process of archiving design documents for the data marts to discuss and share them within the group.

Thanks to this solid process, we do maintain a more organized data environment, but documenting every single business logic and column for a mart has always been a highly labor-intensive task.

Since the task involved generating a Notion page in a fixed template with the SQL logic and the configuration of the designed (or to-be-designed) mart, I built a skill using Notion mcp to automate this with AI, generating drafts that are over 80% complete.

• Given that the template had many requirements, I provided a separate reference file so it could look at existing docs, and

• I continued to refine the skill with a focus on verifying the consistency of each requirement.

/update-notion-mart-template                                                                                                                                                                                      
                                                                                                                                                                                                                    
  - dbt mart 모델을 Notion "Data Modeling" DB에 문서화하는 스킬                                                                                                                                                     
  - SQL 파일에서 자동 추출: 모델명, 소스 테이블(ref/source), materialization, partition key, unique key, 컬럼 목록                                                                                                  
  - YAML 파일에서 테스트 케이스, manifest에서 DAG 스케줄도 자동 추출                                                                                                                                                
  - GROUP BY 절 분석으로 grain, dimension/fact 컬럼 자동 분류                                                                                                                                                       
  - 코드에서 추출 불가능한 비즈니스 컨텍스트만 사용자에게 질문 (현 상황, 어려움, 모니터링 지표)                                                                                                                     
  - 마트 모델링 템플릿 구조로 Notion 페이지 생성/업데이트 (5개 섹션: Business Process → Grain → Dimensions → Facts → Batch Process)
  - 기존 페이지 있으면 업데이트, 없으면 새로 생성 (Status = "In Progress")

As a result, I was able to create a workflow that instantly generates complex Notion pages from a SQL file and a few prompts.

It's Not 'Click and Done' from the Start...

By collaborating heavily with AI and streamlining various parts, I was able to achieve high quality and speed despite tackling the project alone.

Beyond just experiencing AI's efficiency, this project gave me a chance to reflect on the role of a practitioner. I realized that the gap between someone who simply does assigned tasks and someone who continuously creates added value is going to widen even further.

Also, I believe that a certain amount of meticulous fine-tuning is absolutely essential to create use cases that fit your exact needs. I hope sharing this serves as a meaningful example, and I look forward to seeing more fun and novel use cases in the data field as well.