Jiyoung Seo

JiyoungSeo

I research AI that perceives how people move, reasons with grounded knowledge, and predicts how 3D worlds respond. These are the pieces a world model needs, for games and for robots.

  • M.S. in Artificial Intelligence, Korea University Computer Vision Lab (KUAICV), graduating Feb 2027
  • 5 papers including CVPR 2026, CVPR 2025 and a first-author IEEE Access article
  • B.S. Statistics and Artificial Intelligence, University of Seoul
See the research Email me GitHub
The perceive, reason, predict loop Three research directions arranged as a loop around a world model: perceive motion (Ego-Bi), reason with grounded knowledge (M3KG-RAG), and predict 3D worlds and physics (EditSplat and physical AI). Each node links to the matching research. World model Perceive motion Reason knowledge Predict 3D · physics
Each node opens the research behind it. Colors follow the X, Y, Z axes of a 3D viewport and mark the same three directions throughout the page.

Research

Four projects, one per direction of the loop. Each started from a problem existing data or methods could not answer.

Perceive · motion

Ego-Bi: telling apart actions that differ only in direction

IEEE Access 2026, first author

Tightening and loosening a screw look the same in a single frame. Existing datasets barely contained such bidirectional pairs, so models confused them.

  • Recorded real hand-tool work from the first-person view: 1,200+ sequences, 622,737 frames, 19 fine-grained actions, and wrote the annotation guide.
  • Designed the Bidirectional Motion-Prior (BMP) module, which accumulates rotation and wrist direction from 3D hand joints over time.
+8.96%pTop-1 accuracy on Ego-Bi, 83.15% → 92.11%, with gains on FPHA and H2O as well
Paper
Ego-Bi dataset samples: first-person frames of clamp and unscrew actions with hand type and object labels Architecture of the BMP-based action recognition framework with motion, object and hand-type branches
Dataset samples and the BMP framework.
Reason · knowledge

M³KG-RAG: retrieval that checks its own evidence

CVPR 2026, second author

Audio-visual LLMs given outside knowledge often get worse answers, because single-hop graphs and embedding search pull in facts unrelated to the clip.

  • Built a pipeline in which seven role-specific LLM agents turn raw audio, video and text into a multi-hop knowledge graph with no human labeling.
  • Proposed the core idea of GRASP: keep a retrieved fact only if visual and audio grounding confirm it appears in the clip, then let a small LLM prune what does not help the answer.
  • Set up the LLM-as-a-Judge protocol and hand-scored samples to check the judge.
+9.23LLM-as-a-Judge points over prior work, with win rates up to 86.3% across audio, video and audio-visual QA
Paper
Multi-agent pipeline that extracts, grounds and refines knowledge graph triplets GRASP: modality-wise retrieval followed by visual and audio grounding and selective pruning before the multimodal LLM
The agent pipeline that builds the graph, and GRASP retrieval.
Predict · 3D worlds

EditSplat: text edits that stay consistent from every view

CVPR 2025, third author

Editing a 3D Gaussian Splatting scene with a 2D diffusion model view by view makes the result disagree between viewpoints.

  • The method fuses multi-view information into diffusion guidance and trims Gaussians where attention says the edit must happen.
  • My part: reproduced the baselines, ran the comparisons, selected and checked qualitative results, and built the project page.
PaperProject page
EditSplat overview: multi-view fusion, multi-view fusion guidance in the diffusion network, and attention-guided trimming of 3D Gaussians
Multi-view fusion guidance and attention-guided trimming.
Predict · physics

Recovering what objects are made of, from video

In progress, first author, submission planned for Nov 2026

A world model should know not only how a scene looks but how it reacts when touched. This project estimates hidden material properties from multi-view video of dynamic scenes.

  • Redefined contact between different materials in a differentiable form, after the standard contact handling made training unstable.
  • There are no ground-truth labels, so I built the evaluation from simple cases whose physical quantities are known.
Physical property estimation loop Multi-view video is reconstructed with 4D Gaussians, simulated with contact handling, compared with observations, and the error updates the estimated material properties. Multi-view dynamic video 4DGS reconstruction Simulation multi-material contact Compare sim vs. observed update material properties
Estimation loop, simplified.

Publications

Newest first.

  1. CVPR 2026second author

    M³KG-RAG: Multi-hop Multimodal Knowledge Graph-enhanced Retrieval-Augmented Generation

    H Park, J Seo, J Moon, H Park, W Byeon, S Kim, H Im, J Lee, S Kim

    Paper

  2. IEEE Accessfirst author, 2026

    Egocentric Hand Activity Video Dataset and Bidirectional Motion-Priors for Hand Action Recognition

    J Seo, D Lee, P Lee, J Lee, Y Gil, K Ramani, S Kim

    Paper

  3. arXiv 2025second author

    VAT-KG: Knowledge-Intensive Multimodal Knowledge Graph Dataset for Retrieval-Augmented Generation

    H Park, J Seo, M Jang, H Baek, G Chang, H Park, S Kim

    PaperProject page

  4. arXiv 2025third author, under review

    AV-Surf: Surface-Enhanced Geometry-Aware Novel-View Acoustic Synthesis

    H Baek, H Shin, J Seo, C Kim, S Kim, H Kim, S Kim

    PaperProject page

  5. CVPR 2025third author

    EditSplat: Multi-View Fusion and Attention-Guided Optimization for View-Consistent 3D Scene Editing with 3D Gaussian Splatting

    D Lee, H Park, J Seo, E Park, H Park, H Baek, S Shin, S Kim, S Kim

    PaperProject page

How the research moved

Each step took the question one layer closer to a world that responds to actions.

  1. 2023Agents in games

    Trained reinforcement learning agents in the VizDoom environment.

  2. 2024Audio and speech

    Dialect speech to standard speech, end to end. Then sound for new viewpoints in AV-Surf.

  3. 2024–20253D scenes

    Consistent 3D editing with Gaussian Splatting in EditSplat.

  4. 2025–2026Motion and knowledge

    Egocentric motion in Ego-Bi, grounded multimodal knowledge in VAT-KG and M³KG-RAG.

  5. NowPhysics

    Inferring material properties so a reconstructed scene reacts like the real one.

Projects and awards

Outside the lab, mostly as team lead.

WhenWhatRoleOutcome
Jan 2024AICOSS Hackathon (Dacon): multi-label classification of satellite imagesParticipant7th place, HD Hyundai Site Solution Award. Rebalanced a heavily skewed label set with tempered sigmoid, logit adjustment and ASL, then tuned per-class thresholds for mAP.
Mar–Jul 2024TAVE, computer vision trackPresenterCompared HRNet and Faster R-CNN for 24-point body keypoint detection with 5-fold cross-validation.
Jan–Apr 2024Deep daiv, audio deep learningTeam leadDialect speech recognition, translation to standard Korean with Ko-BART, and speech synthesis with FastSpeech2, evaluated by CER. Code
Sep–Dec 2023NLP course projectTeam leadBuilt 2,400 YouTube subtitle and comment pairs and fine-tuned LLaMA2-7B with PEFT to write context-aware comments.
Sep–Dec 2023Reinforcement learning courseParticipantTrained agents in VizDoom. Code

Notes

I keep study notes on this site.

Computer graphics

Raster images, ray tracing, transformation matrices, viewing, the graphics pipeline, signal processing and surface shading.

Browse categories

3D vision paper reviews

2DGS, SuGaR, RaDe-GS, Mip-NeRF 360, PERSONA, 3D-LLM, Pixie, DrivingGaussian++ and more.

Read the blog