Batch Operations
High-performance bulk operations for adding, updating, deleting, and fetching multiple entities at once.
Why Batch?
Batch operations are 10-100x faster than individual operations, especially with cloud storage:
| Operation | Individual | Batched |
|---|---|---|
| Add 100 entities | ~5s (cloud) | ~200ms |
| Get 20 entities | ~6s (cloud) | ~300ms |
| Create 50 relationships | ~2.5s (cloud) | ~150ms |
addMany()
Add multiple entities in a single optimized operation.
add-many.js
import { Brainy, NounType } from '@soulcraft/brainy'
const brain = new Brainy()
await brain.init()
// Add multiple entities
const result = await brain.addMany({
items: [
{ data: 'Alice Smith', type: NounType.Person, metadata: { dept: 'Engineering' } },
{ data: 'Bob Jones', type: NounType.Person, metadata: { dept: 'Marketing' } },
{ data: 'Project Alpha', type: NounType.Project, metadata: { status: 'active' } }
]
})
console.log(`Added ${result.successful.length} entities`)
console.log(`IDs: ${result.successful}`)
With Progress Tracking
add-many-progress.js
const result = await brain.addMany({
items: largeDataset,
chunkSize: 100, // Process in chunks of 100
parallel: true, // Parallel processing
continueOnError: true, // Don't stop on errors
onProgress: (completed, total) => {
const percent = ((completed / total) * 100).toFixed(1)
console.log(`${percent}% complete (${completed}/${total})`)
}
})
// Check for errors
if (result.failed.length > 0) {
console.log(`${result.failed.length} items failed`)
result.failed.forEach(f => console.log(f.error))
}
batchGet()
Fetch multiple entities by ID in a single call. Eliminates N+1 query problems.
batch-get.js
// Get multiple entities at once
const ids = ['user-1', 'user-2', 'user-3', 'user-4']
const entityMap = await brain.batchGet(ids)
// entityMap is a Map<string, Entity>
ids.forEach(id => {
const entity = entityMap.get(id)
if (entity) {
console.log(`${id}: ${entity.data}`)
} else {
console.log(`${id}: not found`)
}
})
// With vectors (when you need them)
const withVectors = await brain.batchGet(ids, { includeVectors: true })
Use Case: VFS Tree Loading
vfs-batch.js
// Instead of N calls for N children:
const relations = await brain.getRelations({ from: parentId })
const childIds = relations.map(r => r.to)
// Single batch call
const children = await brain.batchGet(childIds)
// 20 children on GCS: 6s -> 300ms (20x faster)
deleteMany()
Delete multiple entities efficiently.
delete-many.js
// Delete by IDs
const result = await brain.deleteMany({
ids: ['old-1', 'old-2', 'old-3']
})
console.log(`Deleted ${result.successful.length} entities`)
// With error handling
const result2 = await brain.deleteMany({
ids: idsToDelete,
continueOnError: true
})
if (result2.failed.length > 0) {
console.log('Some deletions failed:', result2.failed)
}
updateMany()
Update multiple entities in bulk.
update-many.js
const result = await brain.updateMany({
items: [
{ id: 'user-1', metadata: { status: 'active' } },
{ id: 'user-2', metadata: { status: 'active' } },
{ id: 'user-3', data: 'Updated content' }
],
chunkSize: 50,
parallel: true,
onProgress: (completed, total) => {
console.log(`Updated ${completed}/${total}`)
}
})
console.log(`Updated ${result.successful.length} entities`)
relateMany()
Create multiple relationships in one call.
relate-many.js
import { VerbType } from '@soulcraft/brainy'
const relationIds = await brain.relateMany({
items: [
{ from: 'alice', to: 'project-1', type: VerbType.WorksOn },
{ from: 'bob', to: 'project-1', type: VerbType.WorksOn },
{ from: 'alice', to: 'bob', type: VerbType.Mentors },
{ from: 'project-1', to: 'team-1', type: VerbType.PartOf }
]
})
console.log(`Created ${relationIds.length} relationships`)
BatchResult Structure
All batch operations return a consistent result structure:
{
successful: ['id-1', 'id-2', 'id-3'], // IDs of successful operations
failed: [
{ id: 'id-4', error: 'Entity not found' },
{ id: 'id-5', error: 'Invalid data' }
]
}
Best Practices
- Chunk size: Use 50-100 for cloud storage, 500-1000 for local
- Continue on error: Enable for fault tolerance in large operations
- Progress tracking: Always add for operations >1000 items
- Parallel processing: Enable for CPU-bound operations, disable for I/O-bound
See Also
- add() - Single entity operations
- import() - File-based bulk import
- Streaming API - Memory-efficient processing