Feat/remove restart policy never if autoscaler v2 and ray 2.55 or greater by fscnick · Pull Request #4816 · ray-project/kuberay

fscnick · 2026-05-08T14:40:30Z

Why are these changes needed?

Starting from ray 2.55.0, the autoscaler restart is supported. The RestartPolicy is Never might not be necessary.

In this PR, it leverages the rayVersion to determine the version of ray. If the version parsed is failed, it would consider it not valid.

Related issue number

Checks

I've made sure the tests are passing.
Testing Strategy
- Unit tests
- Manual tests
- This PR is not tested :(

Signed-off-by: fscnick <fscnick.dev@gmail.com>

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 31cf894175

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

chatgpt-codex-connector · 2026-05-09T00:43:45Z

+//	rayproject/ray:2.55.0-py310      → "2.55.0"
+//	rayproject/ray:2.55.0@sha256:abc → "2.55.0"
+//	rayproject/ray:2.55                → "2.55"
+var rayImageVersionRegex = regexp.MustCompile(`rayproject/ray:(\d+\.\d+(?:\.\d+)?)`)


Parse Ray version from all supported Ray image names

The new version gate only matches rayproject/ray:<ver> (rayImageVersionRegex), so autoscaler-v2 clusters using other supported Ray images (for example rayproject/ray-ml:* in ray-operator/config/samples/ray-service.stable-diffusion.yaml) always fail parsing and are treated as "version unknown". In those cases the operator/webhook keeps enforcing restartPolicy: Never, which means the intended 2.55+ relaxation never activates even when the image tag is new enough.

Useful? React with 👍 / 👎.

Signed-off-by: fscnick <fscnick.dev@gmail.com>

…rt-policy-never-if-autoscaler-v2

AndySung320 · 2026-05-14T01:41:00Z

+	// Use the headGroupSpec to determine whether the RestartPolicy should be Never or not, since the head pod is the one that runs the autoscaler.
+	// The error is ignored here because the function will return false if there's an error parsing the version.
+	// For example, if rayVersion is empty or unparseable, it considers the feature is not valid.


The comment says "Use the headGroupSpec" but the code actually reads from the cluster-level spec (instance.Spec.RayVersion, IsAutoscalingEnabled, IsAutoscalingV2Enabled), not from HeadGroupSpec. Would it be better updating the comment to reflect that both head and worker templates use the same cluster-level gate to decide RestartPolicy ?

fixed at 5978fd1

AndySung320 · 2026-05-14T03:29:05Z

seems to be pre-existing typo "should have the correct"

AndySung320 · 2026-05-14T03:30:03Z

fixed at 0937f50 along with the above.

AndySung320 · 2026-05-14T06:23:11Z

 		if utils.IsAutoscalingV2Enabled(&instance.Spec) {
 			setAutoscalerV2EnvVars(&podTemplate)
-			podTemplate.Spec.RestartPolicy = corev1.RestartPolicyNever
+			if autoscalerRestartValid, _ := utils.IsRayVersionAtLeast(instance.Spec.RayVersion, utils.MinAutoscalerRestartValidVersion); !autoscalerRestartValid {


The error from IsRayVersionAtLeast is silently ignored here. The safe-default behavior (falling back to Never) makes sense, but if rayVersion is empty or unparseable, the user has no way to tell whether Never was set because the version is too old or because their config is wrong. Would it be worth adding a warning log when the error is non-nil?

fixed at 6be130a along with the similar one in DefaultHeadPodTemplate.

hango880623 · 2026-05-15T05:15:58Z

 			return fmt.Errorf("Currently, SidecarMode doesn't support SubmitterConfig")
 		}

 		if rayJob.Spec.RayClusterSpec.HeadGroupSpec.Template.Spec.RestartPolicy != "" && rayJob.Spec.RayClusterSpec.HeadGroupSpec.Template.Spec.RestartPolicy != corev1.RestartPolicyNever {


Does this restart policy also need a change?

Suggested change

// The error is ignored here because the function will return false if there's an error parsing the version.

// For example, if rayVersion is empty or unparseable, it considers the feature is not valid.

autoscalerRestartValid, _ := IsRayVersionAtLeast(rayJob.Spec.RayClusterSpec.RayVersion, MinAutoscalerRestartValidVersion)

if !autoscalerRestartValid && rayJob.Spec.RayClusterSpec.HeadGroupSpec.Template.Spec.RestartPolicy != "" && rayJob.Spec.RayClusterSpec.HeadGroupSpec.Template.Spec.RestartPolicy != corev1.RestartPolicyNever {

I'm not that certain about this. This is set to Never because of SidecarMode not the autoscaler.

Signed-off-by: fscnick <fscnick.dev@gmail.com>

cursor

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{Reviewed by Cursor Bugbot for commit 6be130a. Configure here.}

…rt-policy-never-if-autoscaler-v2

Signed-off-by: fscnick <fscnick.dev@gmail.com>

AndySung320 · 2026-05-19T07:56:53Z

 			expectedRestartPolicy: "",
 		},
-		"Pod template with autoscaling v1 enabled should the correct autoscaler v1 fields": {
+		"Pod template with autoscaling v1 enabled should the the correct autoscaler v1 fields": {


minor typo: "have"

fix at f3458fd.

AndySung320 · 2026-05-19T08:45:44Z

 		mergeAutoscalerOverrides(&autoscalerContainer, instance.Spec.AutoscalerOptions)
 		podTemplate.Spec.Containers = append(podTemplate.Spec.Containers, autoscalerContainer)

+		// The error is ignored here because the function will return false if there's an error parsing the version.


since we keep err here, i think we should also re-word the comment?

AndySung320 · 2026-05-19T08:45:57Z

 	}

+	// Use the RayVersion and autoscaler version to determine whether the RestartPolicy should be Never or not, since the head pod is the one that runs the autoscaler.
+	// The error is ignored here because the function will return false if there's an error parsing the version.


fixed at 565d9a8 along with the above.

Signed-off-by: fscnick <fscnick.dev@gmail.com>

fscnick added 2 commits May 7, 2026 22:27

[RayCluster][autoscaler] not enforce Never if ray is at least 2.55.0

7cad015

Signed-off-by: fscnick <fscnick.dev@gmail.com>

[RayCluster][autoscaler] add test cases

31cf894

Signed-off-by: fscnick <fscnick.dev@gmail.com>

fscnick force-pushed the feat/remove-restart-policy-never-if-autoscaler-v2 branch from 51e5eb6 to 31cf894 Compare May 8, 2026 15:30

fscnick marked this pull request as ready for review May 9, 2026 00:40

fscnick requested review from MortalHappiness, andrewsykim, kevin85421 and rueian as code owners May 9, 2026 00:40

chatgpt-codex-connector Bot reviewed May 9, 2026

View reviewed changes

fscnick marked this pull request as draft May 9, 2026 02:24

fscnick added 2 commits May 11, 2026 21:36

[RayCluster][autoscaler] parse version from rayVersion instead

715c260

Signed-off-by: fscnick <fscnick.dev@gmail.com>

Merge remote-tracking branch 'upstream/master' into feat/remove-resta…

797624a

…rt-policy-never-if-autoscaler-v2

fscnick marked this pull request as ready for review May 11, 2026 15:59

AndySung320 reviewed May 14, 2026

View reviewed changes

hango880623 reviewed May 15, 2026

View reviewed changes

fscnick added 3 commits May 16, 2026 09:21

[RayCluster] fix comment

5978fd1

Signed-off-by: fscnick <fscnick.dev@gmail.com>

[RayCluster] fix test description

0937f50

Signed-off-by: fscnick <fscnick.dev@gmail.com>

[RayCluster] add warning log for rayVersion parsing error

6be130a

Signed-off-by: fscnick <fscnick.dev@gmail.com>

cursor Bot reviewed May 16, 2026

View reviewed changes

Comment thread ray-operator/controllers/ray/common/pod.go Outdated

fscnick added 2 commits May 16, 2026 09:53

Merge remote-tracking branch 'upstream/master' into feat/remove-resta…

2f8947f

…rt-policy-never-if-autoscaler-v2

[RayCluster] avoid noisy log when AutoscalerV2 is not enabled

a309e2a

Signed-off-by: fscnick <fscnick.dev@gmail.com>

AndySung320 reviewed May 19, 2026

View reviewed changes

fscnick added 2 commits May 19, 2026 21:15

[RayCluster] fix test description

f3458fd

Signed-off-by: fscnick <fscnick.dev@gmail.com>

[RayCluster] fix comment

565d9a8

Signed-off-by: fscnick <fscnick.dev@gmail.com>

Conversation

fscnick commented May 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why are these changes needed?

Related issue number

Checks

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot May 9, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

AndySung320 May 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

fscnick commented May 8, 2026 •

edited

Loading

AndySung320 May 14, 2026 •

edited

Loading