MarketGen: A Scalable Simulation Platform with Auto-Generated Embodied Supermarket Environments

1The Hong Kong Polytechnic University 2University of Science and Technology Beijing 3NLPR, MAIS, Institute of Automation, Chinese Academy of Sciences 4University of Chinese Academy of Sciences 5XYZ Embodied AI 6Shandong University 7Linketic
Project Leader Corresponding Author
MarketGen Platform Overview
MarketGen features as a scalable simulation platform with auto-generated scenes for supermarket scenarios. It differs from previous platforms and methods: (b) Handcrafted supermarket scenes in GRUtopia, (c) Tabletop task generation and (d) Rule-based and LLM-based household scene generation.

Abstract

The development of embodied agents for complex commercial environments is hindered by a critical gap in existing robotics datasets and benchmarks, which primarily focus on household or tabletop settings with short-horizon tasks. To address this limitation, we introduce MarketGen, a scalable simulation platform with automatic scene generation for complex supermarket environments. MarketGen features a novel agent-based Procedural Content Generation (PCG) framework. It uniquely supports multi-modal inputs (text and reference images) and integrates real-world design principles to automatically generate complete, structured, and realistic supermarkets. We also provide an extensive and diverse 3D asset library with a total of 1100+ supermarket goods and parameterized facilities assets. Building on this generative foundation, we propose a novel benchmark for assessing supermarket agents, featuring two daily tasks in a supermarket: (1) Checkout Unloading: long-horizon tabletop tasks for cashier agents, and (2) In-Aisle Item Collection: complex mobile manipulation tasks for salesperson agents. We validate our platform and benchmark through extensive experiments, including the deployment of a modular agent system and successful sim-to-real transfer. MarketGen provides a comprehensive framework to accelerate research in embodied AI for complex commercial applications.

Automatic Scene Generation

Agent-based Layout Generation
The Pipeline of Automatic Scene Generation. The agent system first generates a structured spatial layout and semantic info from the input text and reference image. Then the PCG workflow will finish scene construction.

PCG Workflow

With the layout serving as the blueprint for the overall scene structure, the PCG system automatically instantiates and configures these components. This process consists of three primary stages: Asset Retrieval, Adjustment, and Placement.

Drawing from analyses of real-world supermarket facilities, we deconstruct key infrastructure, particularly shelving systems, into their minimal constituent units. By programmatically adjusting parameters, such as the number of vertical tiers, the spacing, and the unit's depth and length, the system can dynamically assemble a wide variety of shelving configurations

PCG Workflow
PCG Workflow with parameterized facilities.

Scene Comparison

Scene Generation Results
Results from our automatic scene generation pipeline. We can achieve a level of fidelity and logical coherence comparable to handcrafted scenes in GRUtopia.

MarketGen Benchmark

To evaluate the capabilities of embodied robots within supermarket environments, we establish a dedicated benchmark primarily focused on long-horizon manipulation tasks. The design of this benchmark is grounded in practical applications, drawing inspiration directly from the daily operational duties performed by human supermarket staff.

Benchmark
An overview of our benchmark. There are two tracks: Checkout Unloading for tabletop manipulation tasks and In-Aisle Item Collection for mobile manipulation tasks.

Experiments

Scene Generation

Layout Generation

Different Combination of Facilities

Auto Product Fulling with different goods categories

Benchmark Experiments

Benchmark Experiments
Qualitative results for two benchmark tracks.

Video demonstration of benchmark experiments.

Additional Generated Scenes

Additional Generated Scenes
More visualization results of the generated supermarkets.

BibTeX

@article{YourPaperKey2024,
  title={Your Paper Title Here},
  author={First Author and Second Author and Third Author},
  journal={Conference/Journal Name},
  year={2024},
  url={https://your-domain.com/your-project-page}
}