Adds USD-level randomization to event manager #1165

Mayankm96 · 2024-10-06T19:15:24Z

Description

Certain scene-level randomizations (such as randomizing the scale) must happen before the simulation starts playing. To this end, the MR adds a new event mode called "scene," which gets called right after the scene design is complete.

Since the event manager CANNOT be initialized before the simulation starts playing, the current implementation hacks the event mode into the environment constructor directly. This is not ideal but at this point, there is no other way around it.

Type of change

New feature (non-breaking change which adds functionality)

Screenshots

If you execute:

./isaaclab.sh -p source/standalone/tutorials/03_envs/create_cube_base_env.py --num_envs 32

Output:

Checklist

I have run the pre-commit checks with ./isaaclab.sh --format
I have made corresponding changes to the documentation
My changes generate no new warnings
I have added tests that prove my fix is effective or that my feature works
I have updated the changelog and the corresponding version in the extension's config/extension.toml file
I have added my name to the CONTRIBUTORS.md or my name already exists there

Mayankm96 · 2024-10-07T12:16:48Z

I already spent a day trying to figure out multiple ways to initialize the event manager before the sim starts playing. The scene entity config resolution is impossible until physics parses the scene (since we need the PhysX body and joint ordering). It is rather unfortunate, but that's how it is right now.

The functionality still works, so if it makes sense, we can move forward with it.

Signed-off-by: Mayank Mittal <[email protected]>

taochenshh · 2024-11-23T01:33:49Z

@Mayankm96 thanks for the great work. wonder if you have plans to merge these functions into the main branch?

kellyguo11 · 2024-12-09T14:56:38Z

source/extensions/omni.isaac.lab/omni/isaac/lab/envs/mdp/events.py

+
+    .. attention::
+        Since this function modifies USD properties, it should only be used before the simulation starts
+        playing. This corresponds to the event mode named "spawn". Using it at simulation time, may lead to


should we add a check for simulation running and throw an error or warning if it's used while simulation is running?

kellyguo11 · 2024-12-09T14:57:10Z

source/extensions/omni.isaac.lab/omni/isaac/lab/envs/mdp/events.py

+            prim_spec = Sdf.CreatePrimInLayer(stage.GetRootLayer(), prim_paths[env_id])
+
+            # get the attribute to randomize
+            scale_spec = prim_spec.GetAttributeAtPath(prim_paths[env_id] + ".xformOp:scale")


what happens if xformOp:scale is not on the prim?

kellyguo11 · 2024-12-09T14:58:37Z

source/extensions/omni.isaac.lab/omni/isaac/lab/envs/mdp/events.py

+            prim_spec = Sdf.CreatePrimInLayer(stage.GetRootLayer(), prim_paths[env_id])
+
+            # get the attribute to randomize
+            color_spec = prim_spec.GetAttributeAtPath(


would it make sense to also support DisplayColor attribute? primitive shapes might not always have materials assigned.

kellyguo11 · 2024-12-09T15:00:23Z

source/extensions/omni.isaac.lab/omni/isaac/lab/envs/mdp/events.py

+            color_spec = prim_spec.GetAttributeAtPath(
+                prim_paths[env_id] + "/geometry/material/Shader.inputs:diffuseColor"
+            )
+            color_spec.default = Gf.Vec3f(*rand_samples[i])


maybe we can handle missing attributes more gracefully than just erroring out. I feel like printing a message and continuing might be more user friendly

jsmith-bdai

Overall, LGTM, but I think we can consolidate the apply_randomization method as it looks identical.

Also, would be good to add a unit test for this functionality

jsmith-bdai · 2024-12-13T16:09:00Z

source/extensions/omni.isaac.lab/omni/isaac/lab/envs/manager_based_env.py

@@ -116,6 +116,10 @@ def __init__(self, cfg: ManagerBasedEnvCfg):
            self.scene = InteractiveScene(self.cfg.scene)
        print("[INFO]: Scene manager: ", self.scene)

+        # randomization at scene level


Can you add a note here why this is needed here for future reference?

jsmith-bdai · 2024-12-13T16:17:05Z

source/extensions/omni.isaac.lab/omni/isaac/lab/envs/manager_based_env.py

+        applied_scene_randomization = False
+        # iterate over all event terms
+        for term_name, term_cfg in self.cfg.events.__dict__.items():
+            # check for non config


Suggested change

# check for non config

# check for none config

jsmith-bdai · 2024-12-13T16:21:03Z

source/extensions/omni.isaac.lab/omni/isaac/lab/envs/direct_marl_env.py

@@ -563,6 +568,39 @@ def set_debug_vis(self, debug_vis: bool) -> bool:
    Helper functions.
    """

+    def _apply_scene_randomization(self):


Can we make this a static helper method so that all 3 implementations can share it? It would need to take in an env of any of the three types and do the same operation on each

jsmith-bdai · 2024-12-13T16:24:13Z

source/extensions/omni.isaac.lab/omni/isaac/lab/envs/mdp/events.py

+        unpredictable behaviors.
+
+    .. note::
+        When randomizing the scale of individual assets, please make sure to set


Should we just throw an exception if replicate_physics is True here?

jsmith-bdai · 2024-12-13T16:24:51Z

source/extensions/omni.isaac.lab/omni/isaac/lab/managers/event_manager.py

@@ -38,7 +38,8 @@ class EventManager(ManagerBase):

    For a typical training process, you may want to apply events in the following modes:

-    - "startup": Event is applied once at the beginning of the training.
+    - "scene": Event is applied once after designing the scene.
+    - "startup": Event is applied once after the simulation starts playing.


Double check that this doesn't need to be updated in any tutorials or how-tos

jsmith-bdai · 2024-12-13T16:25:56Z

source/extensions/omni.isaac.lab/omni/isaac/lab/managers/manager_base.py

+                # check if sim is playing and resolve the entity
+                # joint and body names are only available once the sim is playing
+                # hence, we do not resolve the entity if the sim is not playing


When do the entities get resolved if the sim is not playing?

Mayankm96 added 4 commits October 6, 2024 20:47

fix run instructions

4ca3294

adds event for randomization

dc7dd31

adds scene level randomizations

e0caec0

runs formatter

89df3ec

Mayankm96 changed the title ~~Adds scene-level randomization to event manager~~ Adds USD-level randomization to event manager Oct 6, 2024

Mayankm96 added 2 commits October 6, 2024 21:40

adds better docs

8f24afa

adds support for classes

d3b0605

Mayankm96 requested a review from renezurbruegg October 7, 2024 07:51

Mayankm96 force-pushed the feature/spawn-events branch from 32db208 to d3b0605 Compare October 7, 2024 09:03

Mayankm96 added 4 commits October 7, 2024 12:02

removes unnecessary block

b19b3b1

makes function for scene term parsing

ca136a8

makes a static method to simplify config parsing

98fd020

fixes format

a120c53

Mayankm96 marked this pull request as ready for review October 7, 2024 12:12

Mayankm96 requested review from jsmith-bdai, pascal-roth, kellyguo11 and Dhoeller19 as code owners October 7, 2024 12:12

fix direct task check

870ce8a

Mayankm96 self-assigned this Oct 7, 2024

Mayankm96 added the enhancement New feature or request label Oct 7, 2024

Mayankm96 mentioned this pull request Oct 9, 2024

Adds support to spawn different assets in environments and spawn randomizations #895

Closed

7 tasks

Merge branch 'main' into feature/spawn-events

70d988b

Signed-off-by: Mayank Mittal <[email protected]>

Dhoeller19 unassigned Mayankm96 Oct 31, 2024

kellyguo11 reviewed Dec 9, 2024

View reviewed changes

jsmith-bdai approved these changes Dec 13, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adds USD-level randomization to event manager #1165

Adds USD-level randomization to event manager #1165

Mayankm96 commented Oct 6, 2024 •

edited

Loading

Mayankm96 commented Oct 7, 2024

taochenshh commented Nov 23, 2024

kellyguo11 Dec 9, 2024

kellyguo11 Dec 9, 2024

kellyguo11 Dec 9, 2024

kellyguo11 Dec 9, 2024

jsmith-bdai left a comment

jsmith-bdai Dec 13, 2024

jsmith-bdai Dec 13, 2024

jsmith-bdai Dec 13, 2024

jsmith-bdai Dec 13, 2024

jsmith-bdai Dec 13, 2024

jsmith-bdai Dec 13, 2024

Adds USD-level randomization to event manager #1165

Are you sure you want to change the base?

Adds USD-level randomization to event manager #1165

Conversation

Mayankm96 commented Oct 6, 2024 • edited Loading

Description

Type of change

Screenshots

Checklist

Mayankm96 commented Oct 7, 2024

taochenshh commented Nov 23, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jsmith-bdai left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Mayankm96 commented Oct 6, 2024 •

edited

Loading