v0.1.0 ยท 8 vision cell kinds

An image is a cell. Vision is a formula.

Upload a photo. Get a caption, tags, and detected objects. The same cell kind runs the same way on local models (CLIP, BLIP-2, YOLO) or remote (GPT-4V, Cloud Vision).

๐Ÿฑ
๐ŸŒ…
๐Ÿ™๏ธ
๐Ÿ•
๐Ÿ“ caption vision.caption
upload an image to see the caption
๐Ÿท tags vision.tags
upload an image to see the tags
๐Ÿ” detected objects vision.objects
upload an image to see detected objects
The cell shape

Vision as cells

cells: - id: photo.input kind: value value: "morning.jpg" - id: caption kind: vision input: photo.input model: blip-2 kind: caption - id: tags kind: vision input: photo.input model: clip kind: tags - id: journal.entry kind: formula expr: '"Today I saw: " + caption'

A vision cell is a value cell whose value is the result of running a model on an input image cell. The model can be local (WebGPU) or remote (HTTPS).

8 cell kinds

caption, text (OCR), faces, objects, tags, embed, segment, depth. Each is a vision operation expressed as a cell. The cell's model field says which model to use. The cell's value is the model's output.

  • caption: natural-language description (BLIP-2, LLaVA, GPT-4V)
  • text: OCR โ€” extract all text (Tesseract, Cloud Vision)
  • faces: face detection + landmarks (MTCNN, MediaPipe)
  • objects: object detection (YOLOv8, DETR)
  • tags: descriptive tags (CLIP, RAM)
  • embed: vector embedding for similarity search
  • segment: pixel-level segmentation (SAM)
  • depth: depth map per pixel (MiDaS, Depth Anything)

Models that run anywhere

The same cell shape runs against any model backend. Pick local for privacy, speed, or offline. Pick remote for accuracy, scale, or capability. The cell doesn't care.

  • Local WebGPU: CLIP, BLIP-2, YOLO via transformers.js / onnxruntime-web
  • Local CPU: tesseract.js for OCR, MediaPipe for face detection
  • Remote: OpenAI GPT-4V, Google Cloud Vision, Replicate, Together
  • Future: dedicated chips for inference (Apple Neural Engine, etc.)
View source โ†’ See all 11 repos