Batch Operations

High-performance bulk operations for adding, updating, deleting, and fetching multiple entities at once.

Why Batch?

Batch operations are 10-100x faster than individual operations, especially with cloud storage:

Operation Individual Batched
Add 100 entities ~5s (cloud) ~200ms
Get 20 entities ~6s (cloud) ~300ms
Create 50 relationships ~2.5s (cloud) ~150ms

addMany()

Add multiple entities in a single optimized operation.

add-many.js
import { Brainy, NounType } from '@soulcraft/brainy'

const brain = new Brainy()
await brain.init()

// Add multiple entities
const result = await brain.addMany({
  items: [
    { data: 'Alice Smith', type: NounType.Person, metadata: { dept: 'Engineering' } },
    { data: 'Bob Jones', type: NounType.Person, metadata: { dept: 'Marketing' } },
    { data: 'Project Alpha', type: NounType.Project, metadata: { status: 'active' } }
  ]
})

console.log(`Added ${result.successful.length} entities`)
console.log(`IDs: ${result.successful}`)

With Progress Tracking

add-many-progress.js
const result = await brain.addMany({
  items: largeDataset,
  chunkSize: 100,          // Process in chunks of 100
  parallel: true,           // Parallel processing
  continueOnError: true,    // Don't stop on errors

  onProgress: (completed, total) => {
    const percent = ((completed / total) * 100).toFixed(1)
    console.log(`${percent}% complete (${completed}/${total})`)
  }
})

// Check for errors
if (result.failed.length > 0) {
  console.log(`${result.failed.length} items failed`)
  result.failed.forEach(f => console.log(f.error))
}

batchGet()

Fetch multiple entities by ID in a single call. Eliminates N+1 query problems.

batch-get.js
// Get multiple entities at once
const ids = ['user-1', 'user-2', 'user-3', 'user-4']
const entityMap = await brain.batchGet(ids)

// entityMap is a Map<string, Entity>
ids.forEach(id => {
  const entity = entityMap.get(id)
  if (entity) {
    console.log(`${id}: ${entity.data}`)
  } else {
    console.log(`${id}: not found`)
  }
})

// With vectors (when you need them)
const withVectors = await brain.batchGet(ids, { includeVectors: true })

Use Case: VFS Tree Loading

vfs-batch.js
// Instead of N calls for N children:
const relations = await brain.getRelations({ from: parentId })
const childIds = relations.map(r => r.to)

// Single batch call
const children = await brain.batchGet(childIds)
// 20 children on GCS: 6s -> 300ms (20x faster)

deleteMany()

Delete multiple entities efficiently.

delete-many.js
// Delete by IDs
const result = await brain.deleteMany({
  ids: ['old-1', 'old-2', 'old-3']
})

console.log(`Deleted ${result.successful.length} entities`)

// With error handling
const result2 = await brain.deleteMany({
  ids: idsToDelete,
  continueOnError: true
})

if (result2.failed.length > 0) {
  console.log('Some deletions failed:', result2.failed)
}

updateMany()

Update multiple entities in bulk.

update-many.js
const result = await brain.updateMany({
  items: [
    { id: 'user-1', metadata: { status: 'active' } },
    { id: 'user-2', metadata: { status: 'active' } },
    { id: 'user-3', data: 'Updated content' }
  ],
  chunkSize: 50,
  parallel: true,

  onProgress: (completed, total) => {
    console.log(`Updated ${completed}/${total}`)
  }
})

console.log(`Updated ${result.successful.length} entities`)

relateMany()

Create multiple relationships in one call.

relate-many.js
import { VerbType } from '@soulcraft/brainy'

const relationIds = await brain.relateMany({
  items: [
    { from: 'alice', to: 'project-1', type: VerbType.WorksOn },
    { from: 'bob', to: 'project-1', type: VerbType.WorksOn },
    { from: 'alice', to: 'bob', type: VerbType.Mentors },
    { from: 'project-1', to: 'team-1', type: VerbType.PartOf }
  ]
})

console.log(`Created ${relationIds.length} relationships`)

BatchResult Structure

All batch operations return a consistent result structure:

{
  successful: ['id-1', 'id-2', 'id-3'],  // IDs of successful operations
  failed: [
    { id: 'id-4', error: 'Entity not found' },
    { id: 'id-5', error: 'Invalid data' }
  ]
}

Best Practices

See Also

Next: Entity Extraction →