We propose a novel 360° scene representation for converting real scenes into stereoscopic 3D virtual reality content with head-motion parallax. Our image-based scene representation enables efficient synthesis of novel views with six degrees-of-freedom (6-DoF) by fusing motion fields at two scales: (1) disparity motion fields carry implicit depth information and are robustly estimated from multiple laterally displaced auxiliary viewpoints, and (2) pairwise motion fields enable real-time flow-based blending, which improves the visual fidelity of results by minimizing ghosting and view transition artifacts.
Based on our scene representation, we present an end-to-end system that captures real scenes with a robotic camera arm, processes the recorded data, and finally renders the scene in a head-mounted display in real time (more than 40 Hz).
Our approach is the first to support head-motion parallax when viewing real 360° scenes. We demonstrate compelling results that illustrate the enhanced visual experience – and hence sense of immersion – achieved with our approach compared to widely-used stereoscopic panoramas.
Panorama has always been the most popular representation of the virtual reality scene. However, both panorama and stereo panorama can not represent the depth information of the scene quite well. Hence, I designed a special kind of image representation based on optical flows to stand for the VR scene and a whole set of solutions from capturing to rendering the VR scene. Here is a demo video on how to capture images using our solution.
In the video above, I combined NEQ6 with a Logitech C920 webcam together to capture images from every possible direction. NEQ6 is highly precise and mainly used for astronomy observation. In our hardware solution, it is used to control the position and orientation of the camera.
In avaChat, you could achieve a wholly new experience of chatting. Users can define their avatars in profile settings in 3D. When talking with others, they can see their own and their friend's avatars on the screen, and these avatars will help to express the chatters' mood with body language and facial expression.
What's more, avaChat can save user's chatting history in a particular way, which helps them keep precious memories. Besides, avaChat can also be built as a cross-platform plugin so as to be used on every popular chat app. AR chatting mode is also provided to see the avatar standing in the real scenes.
What's more | avaChat_Holo
avaChat_Holo is the extension of the avaChat on Microsoft HoloLens. It allows you to communicate with others in your real world! Powered by holographic UI, users may feel immersed in the chat and interact with the talker actively.
Light Field Compression
This is my graduate project done during my Master at Tsinghua University. I designed and developed a series of algorithms and applications to store and present the light field. A set of device used for capturing the light field is also invented. The light field is a kind of container which encapsulate all the information of light from every position and direction. Therefore, it is massive and redundant. What my work focuses on is to decrease the redundancy of the traditional light field and designed a compact representation.
My research is comprised of three main tasks: a. A compact representation of light field; b. A real-time method to render it; c. A set of device to capture the light field so as to store and render it. As the video below shows, a whole progress of capturing the light field, converting it into compressed version and rendering while user interaction are demonstrated:
The algorithm for compression of the light field is mainly inspired by Motion Compensation, which is a common method to store the video files efficiently. The light field can be regarded as a video file in higher dimensional space, so the similar methods can be adopted.
K3SimSearch is a simple Python script as a dictionary in which you can look up a GRE word and find its similar words (not synonyms but visual similarity). It works as a small tool for helping students to prepare for GRE.
This is a showcase about viewing the graph of visually similar GRE words.
Parocam (有劲) is an iOS app which utilize GPUImage and Sensetime face detection and alignment technology. The name of the app comes from Parody Camera. It allows real-time recording with various creation modes, such as face features morphing, background changing, and smart masks. With the help of the app, users can create a great many impressing short videos with their faces.
The app is on AppStore at the present. This work was mainly done during my early semesters in Tsinghua University. It is also one of the complete and comercial projects that I've done.
PlanarSight is a course project for Computational Geometry (CS 7024-0183). The course taught by Dr Deng really impressed me and make me feel like diving to the world of computational geometry. This project is a small game made by some advanced algorithms like constrained Delaunay triangulation and visibility polygon construction.
Here is a snapshot about this game:
This project is open source, you can fork it here.
This is a course project for Computer Graphics and Computer Aided Design. It simply triangulated the planar polygon into several triangles. To implement it with high performance, I read the paper and the open source project, poly2tri.
Here is a snapshot of the application:
This project is open source, you can fork it here.
WebGLBrush is my final project for my bachelor's graduation. It's a pure front-end project based on SculptGL. Inspired by ZBrush, a commercial software which does 3D modeling jobs with the digital sculpting solution, I want to implement a web-based 3D sculpting modeling system as my final project.
It presents a big model as the scene in canvas and allows users' interaction to make sculpture on the model. The algorithm behind will do subdivision on mesh surface and perform the precise movement for points on mesh along their normal direction. Thus an effect of WYSIWYG will be shown to users, and it's a light-weighted solution for the small modification of 3D models.
You can try it here. However, loading a model will be time-consuming.
Imagilar is a real-time image similarity search system on the mobile platform. With the rapid development of mobile intelligent devices and wireless communications, users are gradually changing the way of consuming interesting content from the traditional personal computers to smart phones. Hence, I implemented a brand-new content-based image similarity search system which runs on the mobile platform in real time.
This project is accomplished during my internship in Univeristy of Queensland. I also published a short paper based on the project.
This is a project which I submitted for the competition of Innovation Cup in Software Institute, Nanjing University. My team has created an iOS 3D game based on augmented reality by which the whole virtual scene of the game was established and implanted into reality.
The game is made in Unity3D and integrated with the Vuforia Unity Extension. The logic of the game is quite simple. The only task game players need to accomplish is to shoot bullets to the moles to clear them.
However, hardness exists where players have to keep other monsters on the plane. Thus the secret to success is finding a proper position to shoot, which is also the specularity of fun in AR.
2017.9 - 2018.12
M.S. in Computer Science
Vision and Graphics Track:
Attending courses: Computer Graphics, Machine Learning, Deep Learning for Computer Vision, 3D UI and Augmented Reality, Analysis of Algorithms, etc.
2016.6 - 2016.9
Description:a. Integrate 3D animation and emotion with sensor features into the existing message app; b. Build up a 3D message app which should own publish quality based on Skype SDK
Responsibilities:a. Understand the sensor and 3D related knowledge as well as how to integrate 3D animation and emotion with sensor features into the existing message app; b. Build up a 3D message app based on Skype SDK and move features from existing message app to it; c. Complete the 3D message app creation and get to publish quality.
2014.9 - 2017.7
M.Eng. in Software Engineering
Teaching Assistant: Algorithm Analysis and Design, Tsinghua University, 2015
Description: Working on research concerning data compression of static 4D holograms, preparing for submission to the conference
Research:a. Made surveys about lossy and lossless compression of 2D images, video compression based on motion compensation and Image-Based Rendering; b. Designed and implemented a series of algorithms for compression of static 4D holograms; c. Developed a set of devices for capture and compression of static 4D holograms; d. Adjusted parameters and improved performance for the algorithm according to the experiments.
The First Prize Scholarship from Glodon Co., Ltd., at Tsinghua University, 2015
2013.7 - 2013.9
Description: Developed questionnaire platform using Adobe Flex and Spring Security
Responsibilities:a. Implemented a questionnaires administration platform based on Java EE; b. Designed the functionality of visualizing the flow chart of questionnaires presented by Adobe Flex; c. Built authority and security mechanisms for the questionnaire platform with Spring Security.
University of Queensland
2012.10 - 2013.2
Description: Published a paper concerning a real-time image similarity search system on mobile platform: Imagilar
Research:a. Made surveys about image features detection and descriptor representation such as SIFT, FAST and SURF; b. Designed the algorithm for real-time searching and indexing on mobile platform; c. Implemented an iOS application supporting real-time image similarity search; d. Completed and published a paper on Web Information Systems Engineering (WISE2013).
2010.9 - 2014.7
B.S. in Software Engineering
GPA: 4.37/5.0, Major GPA: 4.51/5.0
The Excellent Undergraduate of Nanjing University, 2014
Excellence in the state level, Innovation and Entrepreneurial Project at Nanjing University, 2013
Full scholarship of the Outstanding Undergraduate International Exchange Program from China Scholarship Council, 2012
C++/iOS dev, M.S. in Computer Science at Columbia University, M.Eng in Software Engineering at Tsinghua University, interested in graphics / vision / VR,AR