Remotion to the rescue

The premise

Paper.js is an incredible open-source vector graphics scripting framework, enabling creatives to craft detailed, dynamic visuals directly within the browser. However, when you're dealing with complex animations that require a lot of computational power, your masterpiece can quickly turn into a performance nightmare. I was recently in this exact predicament, but instead of ditching Paper.js, I looked for a different way to deliver the animation. That's when I discovered the magic of Remotion.

Remotion is a remarkable tool that lets you create videos programmatically in React. Initially, I was skeptical of turning my highly interactive Paper.js creation into a static video. But once I dived in, I found it not only relieved performance issues but also offered other unexpected benefits. Here's how I transformed an expensive Paper.js animation into a high-performance video with Remotion.

First, let's have a look at what I started out with. We are talking about the scroll based animation from the landing page of this website. I already had a working version ready to ship to production, tested it many times on my laptop and it was performing adequately. Until i tried it out on my iPhone 12, which completely broke down under the load. It delivered an unsatisfying slide show of 5 FPS. I asked a couple of friends to try out the page on their machines and most of them complained that it was not fluid and felt janky. Pretty frustrating after having put hours and hours into designing, fine-tuning and polishing the crap out of this. Also it's probably worth considering keeping an eye on performance from time to time if you are developing on Apple silicon like I am - these machines tend to rip through anything you throw at them which can give a false sense of performance.

Shadows on canvas are really slow

I then started trying to pin down the performance bottlenecks. Most of the animation was procedurally generated and therefore parametrized. So i could lower the count of blobs, lines, rays, particles, etc and benchmark on my phone. The results were demoralizing. I had to lower the count of elements so drastically that it completely ruined the design. Then I got sceptical. It couldn't be that a relatively modern phone cant handle to draw more than 5 blobs on a canvas. Something must have been wrong. After several more minutes of tinkering i started modifying visual properties on the paperjs instances. I noticed that the shadowBlur property was responsible for the largest portion of the slow down by far. Removing shadows made everything much faster. But it also drastically reduced the visual appeal. I wasn't happy.

Apple does it with videos

My scroll based animation on the landing page was actually not inspired by Apple, but rather by the Remix framework's landing page. This is also why i did not immediately reach for a video, but rather went through the complexity of translating divs and scaling and fading in paperjs instances based on the user's scroll position on the page. Then i remembered a video about the AirPods Pro landing page, and how it made extensive use of videos that are scrubbed through when the user scrolls down on the page. The problem was that i don't really have a lot of expertise in video editing. That was when I remembered a tool I read about in a tweet probably a year ago. It allowed the user to create videos in react. I didn't remember the name of the tool, so i googled a bit and found my way to the Remotion docs.

Setting up remotion

I decided to just create a new project to try this thing out. This allowed me to basically just follow the steps in their fantastic CLI tool when setting up a new project.

bashCopy
1
npm init video

It then asks you to install some dependency and you can then select a template to go with. There are several, i chose "Tailwind and TypeScript" because i was using Tailwind for this website and i wanted to sidestep the setup. That totally worked.

bashCopy
1
npm start

And the browser opens with a video editing UI like this. The experience is stellar, zero headaches so far.

Obviously, it didnt have my content yet. But it was a simple hello world kind of project doing a small animation that teaches you the core concepts. Basically the working principle of Remotion can be summarized like this:

Your video has 4 parameters: width, height, frameCount and frameRate. Every animation is based on the current frame index. This matched exceptionally well with what I already had. My animation was taking the scroll position in the current section and divided it by the height of the section, yielding a value between 0 and 1 which then serves as the main parameter for all animations. I wrote this little hook so i could reuse this for each section.

typescriptCopy
1
import React from "react";
2

3
export const useScrollPosition = () => {
4
  const ref = React.useRef(null);
5
  const [value, setValue] = React.useState(0);
6
  const [scrollY, setScrollY] = React.useState(0);
7
  const [bbox, setBbox] = React.useState(null);
8

9
  React.useEffect(() => {
10
    function scrollListener() {
11
      const bbox = ref.current?.getBoundingClientRect();
12
      if (bbox) {
13
        const percent = Math.min(
14
          1,
15
          Math.max(0, -bbox.y) / (bbox.height - window.innerHeight)
16
        );
17
        setValue(percent);
18
        setScrollY(window.scrollY);
19
        setBbox(bbox);
20
      }
21
    }
22

23
    const bbox = ref.current?.getBoundingClientRect();
24
    if (bbox) setBbox(bbox);
25

26
    window.addEventListener("scroll", scrollListener);
27
    return () => {
28
      window.removeEventListener("scroll", scrollListener);
29
    };
30
  }, []);
31

32
  return { ref, value, scrollY, bbox };
33
};

So to make my animation "move" in Remotion, I simply had to do this:

typescriptCopy
1
const frame = useCurrentFrame();
2
const { durationInFrames } = useVideoConfig();
3
const value = frame / durationInFrames;
4
The rest was just working exactly the same as before. Most of the scenes in the animation are setup like this:
5

6
import React, { useEffect, useRef } from 'react';
7
import paper from 'paper';
8

9
const Animation = () => {
10
  const canvasRef = useRef();
11
  const instancesRef = useRef({});
12

13
  const value = ... // get the fraction of animation progress - see snippet above
14

15
  // this effect runs whenever the value changes, so for every frame of the video
16
  useEffect(() => {
17
    const { path, ... } = instancesRef.current; // get the Paper.js instances
18
    if (!path) return;
19
    // make changes to their properties based on `value`. Here I am just setting a path's
20
    // opacity value equal to the value. This means it will be completely transparent
21
    // at the beginning of the video and completely opaque at the end.
22
    path.opacity = value;
23
  }, [value]);
24

25
  useEffect(() => {
26
    paper.setup(canvasRef.current);
27
    // Create Paper.js instances here and store them in the instancesRef
28
  }, []);
29

30
  return <canvas ref={canvasRef} />;
31
};

The overall experience of making this happen was just extremely satisfying because everything just fell together exactly like it should. There was very little effort involved in getting from a browser based animation that works with canvas and other DOM elements to a video. Certainly some of the smoothness of this transition has to be attributed to the fact that my animation was already computed in JavaScript, i.e there were no CSS transitions involved - these are not supported in Remotion. But I also want to emphasize that Remotion provides an exceptionally stellar developer experience that is second to none. The "Remotion Studio" web app is very clean and gives you the necessary tools to scrub through and monitor your animation while you are writing it. It automatically refreshes when files are saved, so again top notch DX right there. Rendering is also easy, there is both a CLI command and a UI menu for it - i used the UI menu since i was already there and it was easy to find.

Replacing the animation with the video

I placed the video into the public resources directory of my site which makes Remix serve it as a resource. Then i could already request it from the page with a video tag and play it using the controls. But i still had to set it up so that the user's scrolling interaction can drive scrubbing through the video. Here is my implementation of a background video component.

typescriptCopy
1
const BackgroundVideo = React.memo(
2
  ({
3
    src,
4
    className,
5
    value,
6
    start = 0,
7
    end = 1,
8
    visibleBeforeStart = false,
9
    visibleAfterEnd = false,
10
    style,
11
  }: {
12
    src: string;
13
    value: number;
14
    start?: number;
15
    end?: number;
16
    visibleBeforeStart?: boolean;
17
    visibleAfterEnd?: boolean;
18
    className?: string;
19
    style?: React.CSSProperties;
20
  }) => {
21
    const ref = React.useRef<HTMLVideoElement>(null);
22
    const valueRef = useValueRef(lerp(value, start, end));
23

24
    React.useEffect(() => {
25
      const video = ref.current;
26
      if (!video) return;
27

28
      video.play();
29
      video.pause();
30

31
      setInterval(function () {
32
        video.pause();
33
        const t = valueRef.current * video.duration;
34
        video.currentTime = t;
35
      }, 40);
36
    }, []);
37

38
    function getVisible() {
39
      if (value > start && value < end) {
40
        return true;
41
      }
42
      if (value <= start && visibleBeforeStart) {
43
        return true;
44
      }
45
      if (value >= end && visibleAfterEnd) {
46
        return true;
47
      }
48
      return false;
49
    }
50

51
    return (
52
      <div
53
        className={
54
          "fixed top-0 left-0 right-0 bottom-0 overflow-hidden -z-10 " +
55
          className
56
        }
57
        style={{ display: getVisible() ? "block" : "none", ...style }}
58
      >
59
        <video
60
          ref={ref}
61
          playsInline
62
          preload="auto"
63
          muted
64
          className="absolute top-0 left-0 w-full h-full object-cover"
65
        >
66
          <source src={src} type="video/mp4" />
67
        </video>
68
      </div>
69
    );
70
  }
71
);

This is where the only annoyance during this entire process happened. The video was very janky when scrolling. It would sometimes skip half the frames and just jump to the end. Overall a very bad user experience. I first tried a couple of things in the code but nothing really helped. Then I re-rendered the video with a trillion different combinations of codec settings until I finally stumbled over a forum post that gave me one that worked:

shellCopy
1
ffmpeg -i in.mp4 -vf scale=1920:1080 -movflags faststart -vcodec libx264 -crf 23 -g 1 -pix_fmt yuv420p out.mp4

With this, the videos started to play along nicely 🎉