One of the features I've been thinking about (and requested) for a long time was building a universal comics OCR scanner that works on any comics website, not just providers I manually integrate.
I wanted it to work with:
- open or paywalled websites
- vertical or horizontal readers
- websites using any kind of image rendering
- Safari
Last month I finally implemented it.
Sounds straightforward, right?
- Extract the images with the extension
- Send them to the backend for OCR
- Return the results and layer the OCR results
First, you need to "find" the panels across multiple possible layouts:
- vertical scrolling readers (easiest) where all images load at once
- vertical scrolling readers that only render the currently visible image
- horizontal readers that only render the current page
Except, you do not want it to detect images that are not actual comics panels (icons, ads, pixel trackers, headers, thumbnails, avatars), etc.
So I ended up building a DOM scanner using MutationObserver that searches 'img' or 'canvas' elements, then filter them through heuristics (manga images typically have a rectangular aspect ratio, are at least 500 pixels, have more than 1 on the page, etc) to avoid capturing irrelevant images.
So detecting the image elements was fairly straightforward
Extracting data from images
There are several different types of images to handle:
**single origin img ** Easy to extract—just get the src, fetch it, done.
single origin canvas Easy to extract—call toDataUrl(), done.
cross-origin img
If an image is served from a different domain as the webpage, it is considered cross-origin. Trying to extract the image data or fetching the URL directly from JS will trigger a "Tainted canvas error" if the cors headers are unfriendly. You can't do anything client-side. One option that I had with the extension is to rewrite headers via the backend script, but that has issues on Safari (of course). Could inspect requests, but that requires a background script, which I decided against because it's too problematic.
The solution was to proxy through a backend endpoint that rewrites CORS headers to allow access.
img cross origin + CDN bot detection
Even worse, some image CDN (like Cloudlfare) have a built-in bot detection where they expect specific cookies to be present. Just fetching the image from the backend doesnt' work.
I ended up setting up an endpoint that downloads the image using a third-party virtual stealth browser forwarded with a residential proxy as a fallback to bypass CDN bot detection.
This might get costly, but it mostly works for now.
img cross origin + auth cookie
Even one step further, some websites protect the image serving endpoints with an auth cookie.
I can't do really do anything in this case without severely compromisign security.
I could technically use the cookies permission of thee extension, but I don't want to ask for this permission, risk slowing down store reviews even more, and introduce potential critical vulnerabiltiies.
tainted/cross-origin canvas
This is where things get... more complicated.
For tainted 'img' elements, we saw that the best workaround was 'proxying' the image through the backend to rewrite CORS headers.
Unfortuantely, canvas elements do not have an URL that you can forward just like that.
The first thing I tried was spoofing the method that allows the page to create a canvas, to capture the original URL fed in setting up the canvas.
That worked, except.. the image I got was scrambled—likely "atlas" image textures used by the comics service.
What the user sees is differnet from what this source image is.
I got around this by shimming canvas methods in an injected script from the extension, Good—now we have the source image, but we need to convert it to a usable format.
The way we do that is by constructing a clone canvas and replaying exactly the same operations (translations, repositioning) that the website used to construct the canvas. So we capture those operations as well, get an unprotected image by proxying through our server (which adds the necessary CORS headers), and that gives us the final image.
That worked.
Other approaches I tried
Background script proxy / intercept
Requires too many permissions to work on any website. Background scripts on Safari cause too many problems.
Backend proxy + extract cookies in extension
Would solve everything but requires too many permissions.
Screenshot
I could have used the Screen Capture API to grab screenshots of elements on the page.
Attracting idea, but:
- Only the visible viewport is capturable. Anything off-screen is inaccessible unless the user manually scrolls every panel into view, making it feel slow.
- Matching the screenshot back to an img/canvas is brittle. You have to compensate for devicePixelRatio, zoom, scroll offsets, transforms, and any page-level parallax/animations, or you risk cropping the wrong region.
- The API captures everything the user sees: overlays, popups, other extensions.
- It's throttled (~100 ms) and heavy. Doing it on every scroll (even debounced) is far more expensive than our current per-image capture.
- From an extension architecture perspective, content scripts can't call it directly—you need to proxy through the background service worker (which we removed because of Safari).
That's the first behind-the-scenes article about a Lexirise feature. Let me know on Discord if you'd like to read more!