Why a critic model is needed?

[xshadowxx]

Thanks for the nice work, but I'm wondering why there is still a critic model here. The GRPO seems eliminates the critic network. Did you compare the current PPO with GRPO to see which works better?

Thanks

[Zeng-WH]

We haven't tried GRPO yet, and haven't compared the two.

[eryk-mazus]

Relevant blog post from the creators of openrlhf:
https://hijkzzz.notion.site/unraveling-rlhf-and-its-variants-engineering-insights#147d9a33ecc9806090f3d5c749d31f05

[Lineark]

Correct me if I'm wrong. I believe in this scenario the critic model, which comes with the training framework, simply means the one that gives the reward. So in terms of R1 and its replications, it's just the rule-based reward system(no neural network involved).

[insomnia1996]

Correct me if I'm wrong. I believe in this scenario the critic model, which comes with the training framework, simply means the one that gives the reward. So in terms of R1 and its replications, it's just the rule-based reward system(no neural network involved).

I think their critic model is the same as the actor model, which means qwen2.5-7b

[donpromax]

Correct me if I'm wrong. I believe in this scenario the critic model, which comes with the training framework, simply means the one that gives the reward. So in terms of R1 and its replications, it's just the rule-based reward system(no neural network involved).

I think their critic model is the same as the actor model, which means qwen2.5-7b

No，the ref model is the same as the actor model. If you read train/openrlhf/cli/train_ppo_ray_box.py

    if args.critic_pretrain and args.colocate_critic_reward:
        assert (
            args.critic_num_nodes == args.reward_num_nodes
            and args.critic_num_gpus_per_node == args.reward_num_gpus_per_node
        ), f"num_nodes and num_gpus_per_node must be the same when colocate critic and reward model."

        bundles = [
            {"GPU": args.critic_num_gpus_per_node, "CPU": args.critic_num_gpus_per_node}
            for _ in range(args.critic_num_nodes)
        ]
        pg = placement_group(bundles, strategy="STRICT_SPREAD")
        ray.get(pg.ready())

    if args.critic_pretrain:
        critic_model = PPORayActorGroup(
            args.critic_num_nodes,
            args.critic_num_gpus_per_node,
            CriticModelRayActor,
            pg=pg,
            num_gpus_per_actor=0.75 if pg else 1,
        )
    else:
        critic_model = None

Since args.critic_pretrain is None, the critic_model will be None. So for one node implementation: 2 gpus for actor, 2 gpus for ref, 2 gpus for vllm engine