Zhijie Cao

Education

  • 上海交通大学

    Shanghai Jiao Tong University

    M.S. in Control Engineering|Sep 2018 - Mar 2021

    Research at the Institute of Image Processing and Pattern Recognition, Department of Automation, covering computer vision and crowd counting in surveillance scenes.

  • 同济大学

    Tongji University

    B.S. in Automation|Sep 2014 - Jun 2018

    Research focused on embedded systems, hardware development, control theory, machine learning, and computer vision.

Experience

  • 字节跳动

    ByteDance · TikTok · Ad Creative

    Ad Algorithm Engineer|Dec 2023 - Present

    Feed Ad Video Refresh Result link

    1. Developed and maintained video refresh strategies: launched Reply-to and selling-point overlay strategies built on multimodal-model content understanding + LLM text generation + hit-video font retrieval. In xid comparison experiments, the Reply-to strategy delivered advv +12.83%, and the selling-point strategy delivered advv +13.56%. Also maintained legacy refresh strategies such as POV copy and CTA copy.
    2. Explored self-improve agent based video strategy selection: built a multi-agent system of post-delivery collection + experience summarization + experience retrieval, where rules-generator-agent summarizes experience from post-delivery data, rules-manage-agent updates and manages the experience library, and decider-agent retrieves experience and selects strategies. Applied to multi-strategy selection, it delivered strategy-selection accuracy +13.33% and pvr +4.92%.

    Search Ad Image Optimization Result link

    1. E-commerce ad cover image selection: to address overly simple hero images that fail to fully express product information, generated new images via product selling-point extraction + text overlay, combined with online EE selection, delivering advv +1.57% and cvr +0.63% for e-commerce mall ads.
    2. Query-based ad image generation: a refresh approach for existing images — an MLLM picks query-relevant images from ad landing pages and video frames, an LLM generates on-image copy from the query and ad information, and the copy is rendered with face-avoiding text overlay. Combined with online keyword recall + relevance selection, it delivered advv +2.6%.

    Search Ad Title Optimization Result link

    Addressing rough user-written titles that fail to match user search intent.

    1. SFT for relevance: collected each ad's historical user queries offline and rewrote titles conditioned on the query and the original ad information. GPT4 produced the rewritten titles, then distilled via qwen2.5 + SFT; on mall product ads in vertical search it delivered overall advv +0.8% and ctr +0.13%, and on open-loop video ads in general search it delivered overall advv +3.5% and ctr +0.7%.
    2. RLHF for CTR: on top of SFT, trained a CTR estimation model as the reward model and applied RLHF for CTR-oriented title rewriting, delivering advv +0.6% and ctr +1.3 on open-loop general search.
    3. RRHF distillation: on top of RLHF, used RRHF to distill multiple RLHF models; the offline reward win rate rose from 0.62 to 0.93, and the launched model gained a further advv +0.4%.

    Search Ad Creative Pipeline

    Built the entire TikTok search ad creative pipeline from 0 to 1 — material generation and management, plus online selection and delivery.

    1. Offline material rewriting pipelines: built daily offline pipelines for title rewriting, cover image optimization, and video-to-image-text generation.
    2. Online selection: re-engineered the online delivery pipeline and, without increasing pipeline latency, enabled relevance selection and EE selection for titles, video covers, and product covers across general search and e-commerce scenarios.
  • 阿里巴巴

    Alibaba · Taobao & Tmall · Live Content Understanding

    Computer Vision Algorithm Engineer|Mar 2021 - Sep 2023

    LLM Film & TV Auto-Clipping

    1. Built an automated algorithm framework for film/TV commentary-style clipping based on large models, combining LLM-based script rewriting + Scenedet scene segmentation + BLIP2 text-to-video retrieval + video post-processing.
    2. To fix inaccurate retrieval on film/TV verticals, collected vertical-domain data and decomposed the task, enabling automatic clipping of one TV episode in about 30 minutes — a large efficiency gain over manual editing. After deployment to the Guangguang video tab, average watch time per user reached 20s+, exceeding the tab baseline of 14s.

    Live Host Action Understanding

    Localized highlight host actions in live streams (detail showcase, makeup try-on, backpack try-on) and applied them to live recommendation and live clipping.

    1. Defined a host behavior taxonomy covering 8 major e-commerce categories with 30+ classes in total.
    2. Built a streaming action recognition algorithm on VideoSwin, reaching mAP 0.43 and recall 0.26 at 90% accuracy.
    3. Improved the streaming analysis with a label-mask VideoSwin + ActionFormer approach, lifting mAP to 0.62 and recall to 0.42; end-to-end training further raised mAP to 0.69 and recall to 0.55. Compared with original highlights, the clipped short videos delivered +6.7% PV consumed per user, +3.2% watch time per user, and +4.4% transactions per user on the Taobao homepage.
    4. Built a live highlight detection algorithm based on the multimodal temporal model UMT and used highlights as the lead-in segment of clipped videos, further gaining +1.7% watch time per PV, +3.8% transaction conversion (ipv_uv), and +3.7% effective play rate.

    Live Info Card

    To improve live-room conversion, proposed an info card carrying brief product information to remind the host of the product being presented.

    1. Stage one combined product detection + OCR + human detection + rule-based matting; it could extract subjects from arbitrary product images, but top-1 acceptability was only 0.668 due to large variance across categories.
    2. Stage two replaced the complex rules with product-image pre-classification + Visual Grounding, raising top-1 acceptability to 0.773 while cutting GPU memory from 4GB to 2GB.
    3. Host adoption of the card reached 40%, and guided transactions delivered +9.18% GMV per user, showing the card's contribution to transaction value.

    Edge-Cloud Hybrid Live Content Understanding

    1. Built an edge-cloud hybrid live content understanding framework combining on-device small models with cloud large models for inference.
    2. Delivered stable on-device algorithm coverage: with 100k+ (10w+) daily live streams, achieved 75%+ coverage of face, voice, product detection, product retrieval, and ASR algorithms across Android, iOS, and PC.
    3. Reduced compute for highlight production: proposed an edge-cloud multimodal highlight production algorithm where object embeddings and ASR are produced on-device and the cloud only runs large-model inference, cutting cloud compute by 7/8 and saving 5.23M RMB (523w) in compute cost.
    4. Reduced compute for live tagging: across live faceid production, empty-scene detection, and live topic tagging, cut cloud compute by around 30%, saving roughly 10M RMB (1000w) per year.
  • 商汤科技

    SenseTime · Internship

    Computer Vision Intern|Aug 2019 - Jun 2020

    1. Iterated on crowd counting models.
    2. Designed novel loss functions for crowd counting and combined detection with keypoints for crowd counting; submitted to ECCV.

Honors & Certificates

Papers

  • GIGAPIXEL-LEVEL IMAGE CROWD COUNTING USING CSRNET|ICME 2019 Workshop
  • Synthetic guided domain adaptive and edge aware network for crowd counting|SCI Journal

Patents

  • Pedestrian counting technology for light-field camera arrays based on deep learning
  • Multi-pedestrian tracking technology for light-field cameras based on deep learning
  • Crowd counting optimization technology based on head-block loss functions
  • Method for building crowd counting models based on generated images, counting method, and system