4.2 KiB
| id | title | status | source_sections | related_topics | key_equations | key_terms | images | examples | open_questions |
|---|---|---|---|---|---|---|---|---|---|
| ai-workloads | AI Workloads and Model Capabilities | established | Web research: NVIDIA newsroom, Dell product page, WCCFTech, Jeff Geerling, ServeTheHome, Tom's Hardware | [gb10-superchip memory-and-storage ai-frameworks multi-unit-stacking] | [model-memory-estimate] | [llm inference fine-tuning quantization fp4 fp8 fp16 parameter-count] | [] | [llm-memory-estimation.md] | [Tokens/sec for Llama 3.3 70B specifically (only 3B and GPT-OSS-120B benchmarked so far) Maximum batch size for inference at various model sizes Fine-tuning performance — how long to SFT a 7B model on this hardware? Stable Diffusion / image generation performance Training from scratch — is it practical for any meaningful model size?] |
AI Workloads and Model Capabilities
The Dell Pro Max GB10 is designed primarily for local AI inference and fine-tuning, bringing capabilities previously requiring cloud or data center hardware to a desktop form factor.
1. Headline Capabilities
- Up to 200 billion parameter models locally (with quantization)
- 1 PFLOP (1,000 TFLOPS) at FP4 precision with sparsity
- Llama 3.3 70B confirmed to run locally (single unit)
- Up to 405B parameter models with two-unit stacking (see multi-unit-stacking)
1a. Measured Performance (T2 Benchmarked)
| Model / Workload | Performance | Precision | Source |
|---|---|---|---|
| Llama 3.2 3B | ~100 tokens/sec | — | Jeff Geerling |
| GPT-OSS-120B | ~14.5 tokens/sec | INT4 | ServeTheHome |
| Llama 3.1 70B | Competitive w/ Ryzen AI Max+ 395 | — | Jeff Geerling |
| HPL (Linpack) FP64 | ~675 GFLOPS | FP64 | Jeff Geerling |
| Geekbench 6 | Comparable to Ryzen AI Max+ 395; trails Apple M3 Ultra | — | Jeff Geerling |
Prompt processing is noted as a particular strength of the system (T2, Jeff Geerling).
INT4 inference on GPT-OSS-120B is roughly equivalent to an RTX 5070's performance (T2, ServeTheHome).
2. Model Size vs. Memory
With 128 GB of unified memory, the system can hold:
| Precision | Bytes/Param | Max Params (approx) | Example Models |
|---|---|---|---|
| FP4 | 0.5 B | ~200B+ | Large quantized models |
| FP8/INT8 | 1 B | ~100B | Llama 3.3 70B, Mixtral |
| FP16 | 2 B | ~50-55B | Medium models at full prec |
| FP32 | 4 B | ~25-28B | Small models, debugging |
Note: Actual usable capacity is less than 128 GB due to OS, KV cache, framework overhead, and activation memory. Estimates assume ~85-90% of memory available for model weights.
3. Primary Use Cases
Local LLM Inference
- Run large language models privately, no cloud dependency
- Interactive chat, code generation, document analysis
- Privacy-sensitive applications (medical, legal, financial)
Fine-Tuning
- Supervised fine-tuning (SFT) of models using NVIDIA NeMo
- LoRA/QLoRA for parameter-efficient fine-tuning of larger models
- Custom domain adaptation
AI Prototyping
- Rapid iteration on model architectures
- Dataset preprocessing with RAPIDS
- Experiment tracking and evaluation
Data Science
- GPU-accelerated analytics with RAPIDS
- Large-scale data processing
- Graph analytics
Gaming (bonus, not primary use case)
Surprisingly, ARM Linux gaming works via FEX (x86-to-ARM translation) + Steam/Proton:
- Cyberpunk 2077: ~100 fps at 1080p, low settings (T2, Jeff Geerling)
- Doom Eternal: ~200 fps (T2, Jeff Geerling)
Not recommended as a gaming machine — this is a development tool, not a GeForce.
4. Target Users
- AI researchers and developers
- Privacy-conscious organizations
- Academic institutions
- AI prototyping teams
- Independent developers building AI applications
Key Relationships
- Compute provided by: gb10-superchip
- Memory constraints: memory-and-storage
- Frameworks used: ai-frameworks
- Scaling beyond single unit: multi-unit-stacking